[issue44895] refleak test failure in test_exceptions

Change by STINNER Victor : Added file: https://bugs.python.org/file50251/strict_refleak_check.patch ___ Python tracker ___ ___

[issue44895] refleak test failure in test_exceptions

Irit Katriel added the comment: I changed it to +for i in range(2): +dash_R_cleanup(fs, ps, pic, zdc, abcs) +support.gc_collect() (loop twice instead of 3 times) and that seems fine too. -- ___ Python tracker

[issue44895] refleak test failure in test_exceptions

STINNER Victor added the comment: > (once I remove the @skip on test_no_hang_on_context_chain_cycle2). Oh. I forgot that the test is still skipped! > With your patch I don't see a leak for refleak.py, but I still see one for > ./python.exe -m test -R 3:3 test_exceptions -m >

[issue44895] refleak test failure in test_exceptions

Irit Katriel added the comment: With your patch I don't see a leak for refleak.py, but I still see one for ./python.exe -m test -R 3:3 test_exceptions -m test_no_hang_on_context_chain_cycle2 -m test_recursion_normalizing_infinite_exception -m test_recursion_in_except_handler -m

[issue44895] refleak test failure in test_exceptions

STINNER Victor added the comment: New changeset 679cb4781ea370c3b3ce40d3334dc404d7e9d92b by Victor Stinner in branch 'main': bpo-44895: libregrtest: refleak check clears types later (GH-28113) https://github.com/python/cpython/commit/679cb4781ea370c3b3ce40d3334dc404d7e9d92b --

[issue44895] refleak test failure in test_exceptions

Change by STINNER Victor : -- keywords: +patch pull_requests: +26554 pull_request: https://github.com/python/cpython/pull/28113 ___ Python tracker ___

[issue44895] refleak test failure in test_exceptions

STINNER Victor added the comment: > I changed dash_R_cleanup to put the sys._clear_type_cache() after > clear_caches() and the leak is gone Aha! The order of function calls in dash_R_cleanup() matters. Calling sys._clear_type_cache() at the end of dash_R_cleanup() perfectly makes sense.

[issue44895] refleak test failure in test_exceptions

Irit Katriel added the comment: Oh, I see now what you meant. I think you're right that it's cleanup. I changed dash_R_cleanup to put the sys._clear_type_cache() after clear_caches() and the leak is gone: @@ -166,9 +166,6 @@ def dash_R_cleanup(fs, ps, pic, zdc, abcs):

[issue44895] refleak test failure in test_exceptions

STINNER Victor added the comment: > It could be a problem with the way _Py_RefTotal is incremented/decremented. It can be a cache which is kept alive and then clear the following iteration. See my previous comment: test leaked [-4, 4, -4, 4, -4, 4, -4, 4, -4, 4, -4, 4, -4, 4, -4, 4, -4, 4,

[issue44895] refleak test failure in test_exceptions

Irit Katriel added the comment: Victor - I am also getting a feeling that this may not be a real leak but rather a bug in the refleaks test bookkeeping. It could be a problem with the way _Py_RefTotal is incremented/decremented. -- ___ Python

[issue44895] refleak test failure in test_exceptions

STINNER Victor added the comment: A variant of this problem: bpo-36560 with a leak 1 memory block per iteration. -- ___ Python tracker ___

[issue44895] refleak test failure in test_exceptions

Irit Katriel added the comment: I think the leak happens when we invoke GC while the recursion limit is exceeded. It goes way if make this change: def recurse_in_body_and_except(): try: recurse_in_body_and_except() except RecursionError as e:

[issue44895] refleak test failure in test_exceptions

STINNER Victor added the comment: You can also play with the clear() method of frame objects and traceback.clear_frames() (which expects a traceback object). -- ___ Python tracker

[issue44895] refleak test failure in test_exceptions

STINNER Victor added the comment: > test leaked [-4, 4, -4] references, sum=-4 If I disable the GC, the script no longer detects "leaks". Same using a GC threshold of 800. Using a GC threshold between 100 and 700, I reproduce the [-4, 4, -4] references leak. Funny bug. --

[issue44895] refleak test failure in test_exceptions

STINNER Victor added the comment: Irit: do you still reproduce the issue using gc.disable()? Or you can try different values to call gc.set_threshold(). You may also try different values in ns.huntrleaks, like: ns.huntrleaks = (3, 20, 'tt_out.py'). With 20 iterations, do you still reproduce

[issue44895] refleak test failure in test_exceptions

Pablo Galindo Salgado added the comment: There is some timing-related race condition at hand because I tried in a Linux machine and a MacBook Pro (intel) laptop and I wasn't able to reproduce after 20 min of the script running :( -- ___ Python

[issue44895] refleak test failure in test_exceptions

Irit Katriel added the comment: > Could you outline how are you reproducing the leaks? Sorry I missed this. Just run the script: iritkatriel@Irits-MBP cpython-1 % ./python.exe refleak.py -- 1 -- beginning 6 repetitions 123456 .. -- 2 -- beginning 6 repetitions 123456 .. -- 3 --

[issue44895] refleak test failure in test_exceptions

Irit Katriel added the comment: I'm using a mac. It typically fails within at most 10-20 iterations. -- ___ Python tracker ___ ___

[issue44895] refleak test failure in test_exceptions

Mark Shannon added the comment: refleak.py seems to run forever for me. Got bored after 6000 iterations. Which O/S and configure options are needed for refleak.py to fail? -- ___ Python tracker

[issue44895] refleak test failure in test_exceptions

Irit Katriel added the comment: This doesn't leak: try: raise B() except B: tb = sys.exc_info()[1].__traceback__ So this isn't about the traceback being part of the cycle. The B() needs to be there too. --

[issue44895] refleak test failure in test_exceptions

Pablo Galindo Salgado added the comment: > But this makes it not leak again, so now we know it's the traceback: The traceback is a very heavy gc object because it contains the frames, and in the locals of the frame is the traceback itself via the exception. This is a well known heavy cycle,

[issue44895] refleak test failure in test_exceptions

Pablo Galindo Salgado added the comment: I am still amazed that I cannot reproduce the leaks on my machine. Maybe I am missing some steps. Could you outline how are you reproducing the leaks? -- ___ Python tracker

[issue44895] refleak test failure in test_exceptions

Irit Katriel added the comment: But this makes it not leak again, so now we know it's the traceback: try: raise B() except B: b = sys.exc_info()[1] b.__traceback__ = None -- ___ Python tracker

[issue44895] refleak test failure in test_exceptions

Irit Katriel added the comment: It's probably not that because this leaks too, and it skips the cleanup: try: raise B() except B: b = sys.exc_info()[1] -- ___ Python tracker

[issue44895] refleak test failure in test_exceptions

Guido van Rossum added the comment: So there's code generated for an except clause with a variable that clears the variable at the end, so that this works: try: ... except Exception as err: ... print(err) # Should fail, since err was cleared when the except clause exited. Maybe there's

[issue44895] refleak test failure in test_exceptions

Irit Katriel added the comment: I'm pretty sure the frame locals is involved. This leaks (not a is being re-raised, not b): - def test_no_hang_on_context_chain_cycle2(): class A(Exception): pass class B(Exception): pass try: try: raise

[issue44895] refleak test failure in test_exceptions

Change by Irit Katriel : -- Removed message: https://bugs.python.org/msg399763 ___ Python tracker ___ ___ Python-bugs-list mailing

[issue44895] refleak test failure in test_exceptions

Irit Katriel added the comment: I'm pretty sure the frame locals is involved. This leaks (note that a is being re-raised, not b): - def test_no_hang_on_context_chain_cycle2(): class A(Exception): pass class B(Exception): pass try: try:

[issue44895] refleak test failure in test_exceptions

Dong-hee Na added the comment: New changeset c2c857b40f226575d64e0d56a759cbd799f51e62 by Dong-hee Na in branch 'main': bpo-44895: Introduce PYTHONDUMPREFSFILE variable for refcount dumping (GH-27767) https://github.com/python/cpython/commit/c2c857b40f226575d64e0d56a759cbd799f51e62

[issue44895] refleak test failure in test_exceptions

Irit Katriel added the comment: But if I also add a gc loop at the beginning of test_recursion_in_except_handler, the leak goes away. def test_recursion_in_except_handler(): while gc.collect(): pass def set_relative_recursion_limit(n): ... So maybe

[issue44895] refleak test failure in test_exceptions

Irit Katriel added the comment: However, if I replace the assignment with a gc.collect() call, it still leaks: def test_no_hang_on_context_chain_cycle2(): while gc.collect(): pass class A(Exception): pass class B(Exception): pass ... I also changed

[issue44895] refleak test failure in test_exceptions

Pablo Galindo Salgado added the comment: >> x = list(range(100)) Is creating a lot of objects, and the gc has a chance to run on every object creation so maybe what's happening there is that you are inadvertently triggering the gc -- ___ Python

[issue44895] refleak test failure in test_exceptions

Irit Katriel added the comment: It's midnight, so just one last weirdness before I turn into a pumpkin: If I add this assignment to x the issue seems to go away (or become less frequent so I don't see it?) But if I assign x to a small constant it still leaks. def

[issue44895] refleak test failure in test_exceptions

Irit Katriel added the comment: I meant refleaks.py, not reflinks.py. -- ___ Python tracker ___ ___ Python-bugs-list mailing list

[issue44895] refleak test failure in test_exceptions

Irit Katriel added the comment: I've reproduced the leak in 3.10 (with the reflinks.py script). In 3.9 the double-recursion segfaults: Fatal Python error: _Py_CheckRecursiveCall: Cannot recover from stack overflow. Python runtime state: initialized -- versions: +Python 3.10

[issue44895] refleak test failure in test_exceptions

Irit Katriel added the comment: (Pdb) test_no_hang_on_context_chain_cycle2.__code__.co_consts[1] (Pdb) sys.getrefcount(test_no_hang_on_context_chain_cycle2.__code__.co_consts[1]) 2 (Pdb) gc.get_referrers(test_no_hang_on_context_chain_cycle2.__code__.co_consts[1]) [] --

[issue44895] refleak test failure in test_exceptions

Irit Katriel added the comment: The leak seems to be due to the nested definitions of the Exception types A and B. It goes away if I move them out of the function so that it's like: class A(Exception): pass class B(Exception): pass def test_no_hang_on_context_chain_cycle2():

[issue44895] refleak test failure in test_exceptions

Irit Katriel added the comment: I've attached an even shorter script (refleak.py) that reproduces the leak. -- Added file: https://bugs.python.org/file50224/refleak.py ___ Python tracker

[issue44895] refleak test failure in test_exceptions

miss-islington added the comment: New changeset 1960409a6dcbd1e3527f02b523bd27df9086dd77 by Miss Islington (bot) in branch '3.10': bpo-44895: skip test_no_hang_on_context_chain_cycle2 until the refleak is fixed (GH-27761)

[issue44895] refleak test failure in test_exceptions

Change by miss-islington : -- pull_requests: +26250 pull_request: https://github.com/python/cpython/pull/27778 ___ Python tracker ___

[issue44895] refleak test failure in test_exceptions

Łukasz Langa added the comment: New changeset 62bc716fde20fcb7b47416c7959be9e66df93212 by Irit Katriel in branch 'main': bpo-44895: skip test_no_hang_on_context_chain_cycle2 until the refleak is fixed (GH-27761)

[issue44895] refleak test failure in test_exceptions

Irit Katriel added the comment: I created issue44917 for the recursion hang because it seems like it's really another bug. -- ___ Python tracker ___

[issue44895] refleak test failure in test_exceptions

Dong-hee Na added the comment: GH-27767 is a tooling patch for extract dump easily. See cpython-1628938551.dump.gz for msg399588. -- ___ Python tracker ___

[issue44895] refleak test failure in test_exceptions

Change by Dong-hee Na : -- pull_requests: +26243 pull_request: https://github.com/python/cpython/pull/27767 ___ Python tracker ___

[issue44895] refleak test failure in test_exceptions

Dong-hee Na added the comment: Here is the dump I extracted. 0:00:00 load avg: 2.89 Run tests sequentially 0:00:00 load avg: 2.89 [1/1] test_exceptions beginning 6 repetitions 123456 .. test_exceptions leaked [6, 6, 6] references, sum=18 test_exceptions leaked [6, 6, 6] memory blocks,

[issue44895] refleak test failure in test_exceptions

Chris Jerdonek added the comment: FYI, I tried myself, and setting PYTHONHASHSEED didn't make the failures deterministic. -- ___ Python tracker ___

[issue44895] refleak test failure in test_exceptions

Pablo Galindo Salgado added the comment: If someone can reproduce more or less reliably, it would be great to know where are the leaked objects. For that, compile CPython with "-with-trace-refs", surround the suspicious test in a loop (test_no_hang_on_context_chain_cycle2 for example) of a

[issue44895] refleak test failure in test_exceptions

Chris Jerdonek added the comment: Last question: might trying different values of PYTHONHASHSEED help? -- ___ Python tracker ___

[issue44895] refleak test failure in test_exceptions

Irit Katriel added the comment: For instance, if memory addresses of objects are different, so objects sort differently in different runs on the same system. -- ___ Python tracker

[issue44895] refleak test failure in test_exceptions

Chris Jerdonek added the comment: "How does this explain it not being non-deterministic on" -> "How does this explain it not being deterministic on" -- ___ Python tracker

[issue44895] refleak test failure in test_exceptions

Chris Jerdonek added the comment: > Maybe it's a very precise threshold which triggers the issue. Between Linux > and macOS, the site module executes different code paths which produce > different GC counters. > Sometimes, the GC must happen in a very precise line, one line later is too >

[issue44895] refleak test failure in test_exceptions

Irit Katriel added the comment: Regarding the hang in msg399539, do we need some special handling of the case were a RecursionError is being raised and its context is another RecursionError? Like a FatalError? -- ___ Python tracker

[issue44895] refleak test failure in test_exceptions

Dong-hee Na added the comment: Oops sorry. Without PR 27746, I was able to reproduce on my macOS environment. test_exceptions leaked [6, 6, 6] references, sum=18 test_exceptions leaked [6, 6, 6] memory blocks, sum=18 test_exceptions failed (reference leak) --

[issue44895] refleak test failure in test_exceptions

Change by Irit Katriel : -- pull_requests: +26237 stage: needs patch -> patch review pull_request: https://github.com/python/cpython/pull/27761 ___ Python tracker ___

[issue44895] refleak test failure in test_exceptions

Dong-hee Na added the comment: > However hang was able to reproduce. I meant msg399539 -- ___ Python tracker ___ ___

[issue44895] refleak test failure in test_exceptions

Dong-hee Na added the comment: >From my macOS, I was not able to reproduce the issue also as like as Victor >and Pablo. repeat 10 ./python.exe -m test -R 3:3 test_exceptions -m test_no_hang_on_context_chain_cycle2 -m test_recursion_normalizing_infinite_exception -m

[issue44895] refleak test failure in test_exceptions

Irit Katriel added the comment: Adding Mark since he worked on recursion recently. -- nosy: +Mark.Shannon ___ Python tracker ___

[issue44895] refleak test failure in test_exceptions

Irit Katriel added the comment: The script below hangs on my mac (it's an extract from test_recursion_in_except_handler). --- import sys count = 0 def main(): def f(): global count count += 1 try: f() except RecursionError: f()

[issue44895] refleak test failure in test_exceptions

Pablo Galindo Salgado added the comment: I have tried to reproduce this on my Linux and Mac machines and I have been unable to do it for the time being in either of them. -- nosy: +pablogsal ___ Python tracker

[issue44895] refleak test failure in test_exceptions

STINNER Victor added the comment: Chris: > Out of curiosity, is the failure deterministic in environments where it > fails? If not, what is the source of the indeterminism -- some kind of race > condition or something else? The GC uses counters and thresholds to decide which collection and

[issue44895] refleak test failure in test_exceptions

Łukasz Langa added the comment: I merged the workaround to get refleak buildbots unstuck for the weekend. I bumped the issue stage back to "needs patch" as the available one isn't a solution. -- stage: patch review -> needs patch ___ Python

[issue44895] refleak test failure in test_exceptions

miss-islington added the comment: New changeset ebc59262349d6901b825ed9101d604e826757262 by Miss Islington (bot) in branch '3.10': bpo-44895: Temporarily add an extra gc.collect() call (GH-27746) https://github.com/python/cpython/commit/ebc59262349d6901b825ed9101d604e826757262 --

[issue44895] refleak test failure in test_exceptions

Irit Katriel added the comment: > what is the source of the indeterminism -- some kind of race condition or > something else? My hunch is it could be timing of GC. But it's hard to say at this point. -- ___ Python tracker

[issue44895] refleak test failure in test_exceptions

Change by miss-islington : -- nosy: +miss-islington nosy_count: 5.0 -> 6.0 pull_requests: +26229 pull_request: https://github.com/python/cpython/pull/27753 ___ Python tracker

[issue44895] refleak test failure in test_exceptions

Łukasz Langa added the comment: New changeset 7bf28cbb4bf37fa6bdfc2d3f8a3939066b3f8f22 by Irit Katriel in branch 'main': bpo-44895: Temporarily add an extra gc.collect() call (GH-27746) https://github.com/python/cpython/commit/7bf28cbb4bf37fa6bdfc2d3f8a3939066b3f8f22 -- nosy:

[issue44895] refleak test failure in test_exceptions

Irit Katriel added the comment: It’s not deterministic. -- ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue44895] refleak test failure in test_exceptions

Chris Jerdonek added the comment: Out of curiosity, is the failure deterministic in environments where it fails? If not, what is the source of the indeterminism -- some kind of race condition or something else? -- nosy: +chris.jerdonek ___ Python

[issue44895] refleak test failure in test_exceptions

Irit Katriel added the comment: My smallest failing example so far: def test_recursion_in_except_handler(self): self.test_no_hang_on_context_chain_cycle2() def set_relative_recursion_limit(n): depth = 1 while True: try:

[issue44895] refleak test failure in test_exceptions

Irit Katriel added the comment: This fails (test_recursion_normalizing_with_no_memory is not needed): ./python.exe -m test -R 3:3 test_exceptions -m test_no_hang_on_context_chain_cycle2 -m test_recursion_normalizing_infinite_exception -m test_recursion_in_except_handler --

[issue44895] refleak test failure in test_exceptions

Change by Irit Katriel : -- keywords: +patch pull_requests: +26224 stage: -> patch review pull_request: https://github.com/python/cpython/pull/27746 ___ Python tracker ___

[issue44895] refleak test failure in test_exceptions

Irit Katriel added the comment: The problem disappears if I add a gc.collect() loop at the beginning of the new test: diff --git a/Lib/test/test_exceptions.py b/Lib/test/test_exceptions.py index 79798ecf05..e0aeac9d10 100644 --- a/Lib/test/test_exceptions.py +++ b/Lib/test/test_exceptions.py

[issue44895] refleak test failure in test_exceptions

Irit Katriel added the comment: > I don't know why you need to kill it. It's not a long test. Ah I see, -F. -- ___ Python tracker ___

[issue44895] refleak test failure in test_exceptions

STINNER Victor added the comment: > I see it on Mac but not windows. Aha, maybe there is something specific on Mac which triggers an exact code path to trigger to bug. I already saw bugs like that. You can try to play with gc.set_threshold() to change how garbage collection is triggered.

[issue44895] refleak test failure in test_exceptions

Irit Katriel added the comment: I don't know why you need to kill it. It's not a long test. -- ___ Python tracker ___ ___

[issue44895] refleak test failure in test_exceptions

STINNER Victor added the comment: > Victor, if the …cycle2 test is hanging then you need the change in PR27626. That's the commit d5c217475c4957a8084ac3f92ae012ece5edc7cb. My main branch is up to date (commit 8ac0886091c27bf4b6bb0a9b571e174b554d31a4), it includes the commit

[issue44895] refleak test failure in test_exceptions

STINNER Victor added the comment: I am not more lucky in the main branch. I ran it for 30 minutes and it didn't fail. $ ./python -m test -R 3:3 test_exceptions -m test_no_hang_on_context_chain_cycle2 -m test_recursion_normalizing_infinite_exception -m test_recursion_in_except_handler -m

[issue44895] refleak test failure in test_exceptions

Irit Katriel added the comment: Victor, if the …cycle2 test is hanging then you need the change in PR27626. -- ___ Python tracker ___

[issue44895] refleak test failure in test_exceptions

Irit Katriel added the comment: I see it on Mac but not windows. Running the test with -v or adding prints makes the leak go away. I’m suspecting that one of the recursion tests fails to clear an exception or something like that. Removing any one of the tests makes it stop failing.

[issue44895] refleak test failure in test_exceptions

STINNER Victor added the comment: I fail to reproduce the issue on Linux on the 3.10 branch. I interrupted the job after 20 minutes: $ ./python -m test -R 3:3 test_exceptions -m test_no_hang_on_context_chain_cycle2 -m test_recursion_normalizing_infinite_exception -m

[issue44895] refleak test failure in test_exceptions

Change by STINNER Victor : -- nosy: +vstinner ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue44895] refleak test failure in test_exceptions

Change by Dong-hee Na : -- nosy: +corona10 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue44895] refleak test failure in test_exceptions

New submission from Irit Katriel : For background see https://bugs.python.org/issue33930#msg399403 iritkatriel@Irits-MBP cpython % repeat 10 ./python.exe -m test -R 3:3 test_exceptions -m test_no_hang_on_context_chain_cycle2 -m test_recursion_normalizing_infinite_exception -m