[issue37788] fix for bpo-36402 (threading._shutdown() race condition) causes reference leak

[Victor Vorobev]

Change by Victor Vorobev :


--
nosy: +victorvorobev
nosy_count: 15.0 -> 16.0
pull_requests: +26659
pull_request: https://github.com/python/cpython/pull/28239

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37788] fix for bpo-36402 (threading._shutdown() race condition) causes reference leak

[Roundup Robot]

Change by Roundup Robot :


--
pull_requests: +25992
pull_request: https://github.com/python/cpython/pull/27473

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37788] fix for bpo-36402 (threading._shutdown() race condition) causes reference leak

[STINNER Victor]

STINNER Victor  added the comment:

Fix in the main branch:

commit c10c2ec7a0e06975e8010c56c9c3270f8ea322ec
Author: Antoine Pitrou 
Date:   Fri May 14 21:37:20 2021 +0200

bpo-37788: Fix reference leak when Thread is never joined (GH-26103)



When a Thread is not joined after it has stopped, its lock may remain in 
the _shutdown_locks set until interpreter shutdown.  If many threads are 
created this way, the _shutdown_locks set could therefore grow endlessly.  To 
avoid such a situation, purge expired locks each time a new one is added or 
removed.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37788] fix for bpo-36402 (threading._shutdown() race condition) causes reference leak

[Antoine Pitrou]

Change by Antoine Pitrou :


--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37788] fix for bpo-36402 (threading._shutdown() race condition) causes reference leak

[miss-islington]

miss-islington  added the comment:


New changeset b30b25b26663fb6070b8ed86fe3a20dcb557d05d by Antoine Pitrou in 
branch '3.9':
[3.9] bpo-37788: Fix reference leak when Thread is never joined (GH-26103) 
(GH-26142)
https://github.com/python/cpython/commit/b30b25b26663fb6070b8ed86fe3a20dcb557d05d


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37788] fix for bpo-36402 (threading._shutdown() race condition) causes reference leak

[miss-islington]

miss-islington  added the comment:


New changeset 71dca6ea73aaf215fafa094512e8c748248c16b0 by Miss Islington (bot) 
in branch '3.10':
[3.10] bpo-37788: Fix reference leak when Thread is never joined (GH-26103) 
(GH-26138)
https://github.com/python/cpython/commit/71dca6ea73aaf215fafa094512e8c748248c16b0


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37788] fix for bpo-36402 (threading._shutdown() race condition) causes reference leak

[Antoine Pitrou]

Change by Antoine Pitrou :


--
pull_requests: +24777
pull_request: https://github.com/python/cpython/pull/26142

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37788] fix for bpo-36402 (threading._shutdown() race condition) causes reference leak

[Antoine Pitrou]

Change by Antoine Pitrou :


--
versions:  -Python 3.8

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37788] fix for bpo-36402 (threading._shutdown() race condition) causes reference leak

[miss-islington]

Change by miss-islington :


--
nosy: +miss-islington
nosy_count: 14.0 -> 15.0
pull_requests: +24772
pull_request: https://github.com/python/cpython/pull/26138

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37788] fix for bpo-36402 (threading._shutdown() race condition) causes reference leak

[Antoine Pitrou]

Change by Antoine Pitrou :


--
nosy: +pitrou
nosy_count: 13.0 -> 14.0
pull_requests: +24745
pull_request: https://github.com/python/cpython/pull/26103

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37788] fix for bpo-36402 (threading._shutdown() race condition) causes reference leak

[Antoine Pitrou]

Change by Antoine Pitrou :


--
versions: +Python 3.11

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37788] fix for bpo-36402 (threading._shutdown() race condition) causes reference leak

[hai shi]

hai shi  added the comment:

I created PR 25226. It's a another way to solve this problem.
Compared to PR 15175, `_set_sentinel()` don't need to receive any params :)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37788] fix for bpo-36402 (threading._shutdown() race condition) causes reference leak

[hai shi]

Change by hai shi :


--
nosy: +shihai1991
nosy_count: 12.0 -> 13.0
pull_requests: +23963
pull_request: https://github.com/python/cpython/pull/25226

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37788] fix for bpo-36402 (threading._shutdown() race condition) causes reference leak

[Martin Panter]

Martin Panter  added the comment:

Anselm's pull request PR 15175 looked hopeful to me. I've been using those 
changes in our builds at work for a while.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37788] fix for bpo-36402 (threading._shutdown() race condition) causes reference leak

[Roundup Robot]

Change by Roundup Robot :


--
nosy: +python-dev
nosy_count: 11.0 -> 12.0
pull_requests: +23895
stage: needs patch -> patch review
pull_request: https://github.com/python/cpython/pull/25148

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37788] fix for bpo-36402 (threading._shutdown() race condition) causes reference leak

[Jason R. Coombs]

Jason R. Coombs  added the comment:

No. The issue remains open.

--
stage:  -> needs patch
versions:  -Python 3.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37788] fix for bpo-36402 (threading._shutdown() race condition) causes reference leak

[Zhipeng Xie]

Zhipeng Xie <775350...@qq.com> added the comment:

ping

I also encountered this problem. Is there a fix now?

--
nosy: +Zhipeng Xie

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37788] fix for bpo-36402 (threading._shutdown() race condition) causes reference leak

[STINNER Victor]

STINNER Victor  added the comment:

Oh, I'm confused. This issue is about the threading module, whereas bpo-37193 
is unrelated: it's about the socketserver module.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37788] fix for bpo-36402 (threading._shutdown() race condition) causes reference leak

[Mark Dickinson]

Mark Dickinson  added the comment:

Re-opening; I believe this issue is still valid.

Here's output on a debug build on current master (commit 
7591d9455eb37525c832da3d65e1a7b3e6dbf613) on my machine:

lovelace:cpython mdickinson$ ./python.exe
Python 3.10.0a6+ (remotes/upstream/master:7591d9455e, Mar 13 2021, 12:04:31) 
[Clang 11.0.0 (clang-1100.0.33.17)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys, threading, time
>>> for _ in range(10):
... threading.Thread().start()
... time.sleep(0.1)
... print(sys.gettotalrefcount())
... 
109901
109905
109907
109909
109911
109913
109915
109917
109919
109921

--
resolution: fixed -> 
stage: resolved -> 
status: closed -> open

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37788] fix for bpo-36402 (threading._shutdown() race condition) causes reference leak

[Mark Dickinson]

Mark Dickinson  added the comment:

@Victor: How does the fix for #37193 solve this issue? I think I'm missing 
something.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37788] fix for bpo-36402 (threading._shutdown() race condition) causes reference leak

[STINNER Victor]

STINNER Victor  added the comment:

bpo-37193 has been fixed.

--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37788] fix for bpo-36402 (threading._shutdown() race condition) causes reference leak

[Mark Dickinson]

Change by Mark Dickinson :


--
nosy: +mark.dickinson

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37788] fix for bpo-36402 (threading._shutdown() race condition) causes reference leak

[Martin Panter]

Change by Martin Panter :


--
keywords: +3.7regression

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37788] fix for bpo-36402 (threading._shutdown() race condition) causes reference leak

[Jason R. Coombs]

Jason R. Coombs  added the comment:

I marked bpo-42263 as a duplicate of this issue.

This issue is implicated in preventing the desired fix for bpo-37193, where a 
thread wishes to remove the handle to itself after performing its duty. By 
removing its own handle, it can never be joined, and thus obviates the most 
straightforward way to directly remove the handle for a thread within that 
thread.

--
nosy: +jaraco

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37788] fix for bpo-36402 (threading._shutdown() race condition) causes reference leak

[Jason R. Coombs]

Change by Jason R. Coombs :


--
versions: +Python 3.10

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37788] fix for bpo-36402 (threading._shutdown() race condition) causes reference leak

[Виктор]

Change by Виктор :


--
nosy: +ViktorPegy

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37788] fix for bpo-36402 (threading._shutdown() race condition) causes reference leak

[Arkady]

Arkady  added the comment:

I have reproduced a similar memory leak in the multiprocessing

This is the trace:

913 memory blocks: 33232 Bytes:  File "/usr/lib/python3.7/threading.py", line 
890;self._bootstrap_inner();  File "/usr/lib/python3.7/threading.py", line 
914;self._set_tstate_lock();  File "/usr/lib/python3.7/threading.py", line 
904;self._tstate_lock = _set_sentinel();

--
nosy: +Arkady M

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37788] fix for bpo-36402 (threading._shutdown() race condition) causes reference leak

[STINNER Victor]

STINNER Victor  added the comment:

I marked bpo-39201 as a duplicate of this issue.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37788] fix for bpo-36402 (threading._shutdown() race condition) causes reference leak

[STINNER Victor]

STINNER Victor  added the comment:

I marked bpo-39074 as a duplicate of this issue.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37788] fix for bpo-36402 (threading._shutdown() race condition) causes reference leak

[Martin Panter]

Change by Martin Panter :


--
nosy: +martin.panter

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37788] fix for bpo-36402 (threading._shutdown() race condition) causes reference leak

[Adam]

Adam  added the comment:

I ran into this bug as well, and opened an issue for it (before I saw this 
issue): https://bugs.python.org/issue39074

Was there a conclusion on the best way to fix this? It seems like the previous 
__del__ implementation would correct the resource leakage by removing the 
_tstate_lock from _shutdown_locks.

--
nosy: +krypticus

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37788] fix for bpo-36402 (threading._shutdown() race condition) causes reference leak

[Pablo Galindo Salgado]

Pablo Galindo Salgado  added the comment:

Hummmis this really a regression? The docs say:

> No other methods (except for the constructor) should be overridden in a 
> subclass. In other words, only override the __init__() and run() methods of 
> this class.

So if someone is overriding join() they are out of contract

--
nosy: +pablogsal

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37788] fix for bpo-36402 (threading._shutdown() race condition) causes reference leak

[STINNER Victor]

STINNER Victor  added the comment:

Copy of wmanley's comment:
https://github.com/python/cpython/pull/13948#issuecomment-541076882
"""
This caused a regression for people overriding Thread.join to implement custom 
thread interruption. See 
https://stackoverflow.com/questions/50486083/ending-non-daemon-threads-when-shutting-down-an-interactive-python-session
"""

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37788] fix for bpo-36402 (threading._shutdown() race condition) causes reference leak

[STINNER Victor]

STINNER Victor  added the comment:


New changeset d11c2c607768fa549b1aed7899edc061b2ebf19f by Victor Stinner in 
branch 'master':
Revert "bpo-37788: Fix a reference leak if a thread is not joined (GH-15228)" 
(GH-15338)
https://github.com/python/cpython/commit/d11c2c607768fa549b1aed7899edc061b2ebf19f


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37788] fix for bpo-36402 (threading._shutdown() race condition) causes reference leak

[STINNER Victor]

STINNER Victor  added the comment:

> https://github.com/python/cpython/commit/d3dcc92778807ae8f7ebe85178f36a29711cd478

This change introduced a regression :-(
https://github.com/python/cpython/pull/15228#issuecomment-522792133

I created PR 15338 to revert it

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37788] fix for bpo-36402 (threading._shutdown() race condition) causes reference leak

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +15055
pull_request: https://github.com/python/cpython/pull/15338

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37788] fix for bpo-36402 (threading._shutdown() race condition) causes reference leak

[STINNER Victor]

STINNER Victor  added the comment:


New changeset d3dcc92778807ae8f7ebe85178f36a29711cd478 by Victor Stinner in 
branch 'master':
bpo-37788: Fix a reference leak if a thread is not joined (GH-15228)
https://github.com/python/cpython/commit/d3dcc92778807ae8f7ebe85178f36a29711cd478


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37788] fix for bpo-36402 (threading._shutdown() race condition) causes reference leak

[miss-islington]

Change by miss-islington :


--
pull_requests: +15054
pull_request: https://github.com/python/cpython/pull/15337

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37788] fix for bpo-36402 (threading._shutdown() race condition) causes reference leak

[miss-islington]

Change by miss-islington :


--
pull_requests: +15053
pull_request: https://github.com/python/cpython/pull/15336

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37788] fix for bpo-36402 (threading._shutdown() race condition) causes reference leak

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +14952
pull_request: https://github.com/python/cpython/pull/15228

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37788] fix for bpo-36402 (threading._shutdown() race condition) causes reference leak

[Kubilay Kocak]

Change by Kubilay Kocak :


--
nosy: +koobs

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37788] fix for bpo-36402 (threading._shutdown() race condition) causes reference leak

[Anselm Kruis]

Change by Anselm Kruis :


--
nosy: +vstinner

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37788] fix for bpo-36402 (threading._shutdown() race condition) causes reference leak

[Anselm Kruis]

Anselm Kruis  added the comment:

The root cause for the reference leak is the global set 
threading._shutdown_locks. It contains Thread._tstate_lock locks of non-daemon 
threads. If a non-daemon thread terminates and no other thread joins the 
terminated thread, the _tstate_lock remains in threading._shutdown_locks 
forever.

I could imagine that a long running server could accumulate many locks in 
threading._shutdown_locks over time. Therefore the leak should be fixed.

There are probably several ways to deal with this issue. A straight forward 
approach is to discard the lock from within `tstate->on_delete` hook, that is 
function "void release_sentinel(void *)" in _threadmodule.c. Pull request 
(GH-15175) implements this idea. Eventually I should add another C-Python 
specific test-case to the PR.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37788] fix for bpo-36402 (threading._shutdown() race condition) causes reference leak

[Anselm Kruis]

Change by Anselm Kruis :


--
pull_requests: +14906
stage:  -> patch review
pull_request: https://github.com/python/cpython/pull/15175

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37788] fix for bpo-36402 (threading._shutdown() race condition) causes reference leak

[Anselm Kruis]

New submission from Anselm Kruis :

Starting with commit 468e5fec (bpo-36402: Fix threading._shutdown() race 
condition (GH-13948)) the following trivial test case leaks one reference and 
one memory block.

class MiscTestCase(unittest.TestCase):
def test_without_join(self):
# Test that a thread without join does not leak references.
# Use a debug build and run "python -m test -R: test_threading"
threading.Thread().start()

Attached is a patch, that adds this test case to Lib/test/test_threading.py. 
After you apply this patch "python -m test -R: test_threading" leaks one 
(additional) reference. This leak is also present in Python 3.7.4 and 3.8.

I'm not sure, if it correct not to join a thread, but it did work flawlessly 
and didn't leak in previous releases.
I didn't analyse the root cause yet.

--
components: Library (Lib)
files: threading-leak-test-case.diff
keywords: patch
messages: 349173
nosy: anselm.kruis
priority: normal
severity: normal
status: open
title: fix for bpo-36402 (threading._shutdown() race condition) causes 
reference leak
type: resource usage
versions: Python 3.7, Python 3.8, Python 3.9
Added file: https://bugs.python.org/file48534/threading-leak-test-case.diff

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com