[issue31217] test_code leaked [1, 1, 1] memory blocks on x86 Gentoo Refleaks 3.6/3.x

STINNER Victor added the comment: Ok, this issue was really insane. Thank you very much Antoine Pitrou for explaning me the root bug ("int - int") and for the fix. The bug is now fixed in Python 3.6 and master (3.7). I checked manually the applied fix on the Gentoo Refleak 3.x buildbot using

[issue31217] test_code leaked [1, 1, 1] memory blocks on x86 Gentoo Refleaks 3.6/3.x

STINNER Victor added the comment: New changeset 98c849a2f32f6727239b4cce38b8f0ff8adeef22 by Victor Stinner in branch '3.6': bpo-31217: Fix regrtest -R for small integer (#3260) (#3261) https://github.com/python/cpython/commit/98c849a2f32f6727239b4cce38b8f0ff8adeef22 --

[issue31217] test_code leaked [1, 1, 1] memory blocks on x86 Gentoo Refleaks 3.6/3.x

Changes by STINNER Victor : -- pull_requests: +3304 ___ Python tracker ___ ___

[issue31217] test_code leaked [1, 1, 1] memory blocks on x86 Gentoo Refleaks 3.6/3.x

STINNER Victor added the comment: New changeset 6c2feabc5dac2f3049b15134669e9ad5af573193 by Victor Stinner in branch 'master': bpo-31217: Fix regrtest -R for small integer (#3260) https://github.com/python/cpython/commit/6c2feabc5dac2f3049b15134669e9ad5af573193 --

[issue31217] test_code leaked [1, 1, 1] memory blocks on x86 Gentoo Refleaks 3.6/3.x

Changes by STINNER Victor : -- pull_requests: +3303 ___ Python tracker ___ ___

[issue31217] test_code leaked [1, 1, 1] memory blocks on x86 Gentoo Refleaks 3.6/3.x

Antoine Pitrou added the comment: > It makes sure that you get a single object in memory for the same integer, > and not only for Python "small integer singletons"? Yes. > About the -1000..1000 range: if a function leaks more than 1 memory block or > more than 1 reference, there is already

[issue31217] test_code leaked [1, 1, 1] memory blocks on x86 Gentoo Refleaks 3.6/3.x

STINNER Victor added the comment: +int_pool = {i: i for i in range(-1000, 1000)} +def get_pooled_int(v): +return int_pool.setdefault(v, v) I'm not sure that I understand this code. It makes sure that you get a single object in memory for the same integer, and not only for

[issue31217] test_code leaked [1, 1, 1] memory blocks on x86 Gentoo Refleaks 3.6/3.x

STINNER Victor added the comment: Ok, I wrote a PR for my str(int(x)) workaround: https://github.com/python/cpython/pull/3258 I tested manually on the buildbot that it fixes all bugs listed on this issue. -- ___ Python tracker

[issue31217] test_code leaked [1, 1, 1] memory blocks on x86 Gentoo Refleaks 3.6/3.x

Changes by STINNER Victor : -- pull_requests: +3302 ___ Python tracker ___ ___

[issue31217] test_code leaked [1, 1, 1] memory blocks on x86 Gentoo Refleaks 3.6/3.x

STINNER Victor added the comment: > So the first non-zero alloc_delta really has a snowball effect, as it creates > new memory block which will produce a non-zero alloc_delta on the next run, > etc. Oh, I suspected an issue around this code but I was unable to explain it. I focused strongly

[issue31217] test_code leaked [1, 1, 1] memory blocks on x86 Gentoo Refleaks 3.6/3.x

Antoine Pitrou added the comment: This patch does the trick: diff --git a/Lib/test/libregrtest/refleak.py b/Lib/test/libregrtest/refleak.py index efe5210..68e490a 100644 --- a/Lib/test/libregrtest/refleak.py +++ b/Lib/test/libregrtest/refleak.py @@ -48,6 +48,11 @@ def dash_R(the_module, test,

[issue31217] test_code leaked [1, 1, 1] memory blocks on x86 Gentoo Refleaks 3.6/3.x

Antoine Pitrou added the comment: Ahah. Actually, it's quite simple :-) On 64-bit Python: >>> id(82914 - 82913) == id(1) True On 32-bit Python: >>> id(82914 - 82913) == id(1) False So the first non-zero alloc_delta really has a snowball effect, as it creates new memory block which will

[issue31217] test_code leaked [1, 1, 1] memory blocks on x86 Gentoo Refleaks 3.6/3.x

STINNER Victor added the comment: Antoine: "I doubt it. If that was the case, the reference count would increase as well." The bug is really weird :-) Antoine: "Actually, leak2.py doesn't attempt to cleanup anything between runs, so it can't be reliable for detecting leaks. You lack

[issue31217] test_code leaked [1, 1, 1] memory blocks on x86 Gentoo Refleaks 3.6/3.x

Antoine Pitrou added the comment: Actually, leak2.py doesn't attempt to cleanup anything between runs, so it can't be reliable for detecting leaks. You lack dash_R_cleanup() somewhere. -- ___ Python tracker

[issue31217] test_code leaked [1, 1, 1] memory blocks on x86 Gentoo Refleaks 3.6/3.x

Antoine Pitrou added the comment: > It seems like the "leak" is the call to sys.getallocatedblocks() which > creates a new integer, and the integer is kept alive between two loop > iterations. I doubt it. If that was the case, the reference count would increase as well. --

[issue31217] test_code leaked [1, 1, 1] memory blocks on x86 Gentoo Refleaks 3.6/3.x

STINNER Victor added the comment: leak2.py: script based on my leak.py and Antoine's test_haypoleak.py. If you replace "if 0:" with "if 1:", it seems like the test doesn't anymore. It seems like the "leak" is the call to sys.getallocatedblocks() which creates a new integer, and the integer is

[issue31217] test_code leaked [1, 1, 1] memory blocks on x86 Gentoo Refleaks 3.6/3.x

Antoine Pitrou added the comment: Attached test script still fails even though the objects are pre-allocated, and without setting a hash seed. $ ./python -m test -R 3:6 test_haypoleak Run tests sequentially 0:00:00 load avg: 0.23 [1/1] test_haypoleak beginning 9 repetitions 123456789 .

[issue31217] test_code leaked [1, 1, 1] memory blocks on x86 Gentoo Refleaks 3.6/3.x

STINNER Victor added the comment: I'm still able to reproduce the bug using attached leak.py script which is now 100 lines long, but has to import logging, socket and pickle. Without these imports, the bug hides (maybe the hardcoded hash values must be recomputed, but I failed to find hash

[issue31217] test_code leaked [1, 1, 1] memory blocks on x86 Gentoo Refleaks 3.6/3.x

STINNER Victor added the comment: I created bpo-31317: Memory leak in dict with shared keys. -- ___ Python tracker ___

[issue31217] test_code leaked [1, 1, 1] memory blocks on x86 Gentoo Refleaks 3.6/3.x

Changes by Zachary Ware : -- nosy: +zach.ware ___ Python tracker ___ ___

[issue31217] test_code leaked [1, 1, 1] memory blocks on x86 Gentoo Refleaks 3.6/3.x

STINNER Victor added the comment: Copy attached test_leak.py to Lib/test/ and run: set -x while true; do PYTHONHASHSEED=$RANDOM ./python -m test -R 3:30 -j1 test_leak || break; done -- Added file: http://bugs.python.org/file47110/test_leak.py ___

[issue31217] test_code leaked [1, 1, 1] memory blocks on x86 Gentoo Refleaks 3.6/3.x

STINNER Victor added the comment: Ok, I succeeded to reproduce the issue on my amd64 laptop using gcc -m32. I'm now sure that the issue can be reproduced on 32-bit. Maybe the issue can be reproduced on 64-bit but is more unlikely? -- ___ Python

[issue31217] test_code leaked [1, 1, 1] memory blocks on x86 Gentoo Refleaks 3.6/3.x

STINNER Victor added the comment: Current status of my analysis. I'm able to reproduce the bug using this modified Lib/test/test_sys.py: --- import unittest, test.support from test.support.script_helper import assert_python_ok import threading def f123(): pass class

[issue31217] test_code leaked [1, 1, 1] memory blocks on x86 Gentoo Refleaks 3.6/3.x

STINNER Victor added the comment: Another command to reproduce a leak: ./python -m test -R 3:3 -j1 test_sys -F -m test.test_sys.SysModuleTest.test_current_frames -- ___ Python tracker

[issue31217] test_code leaked [1, 1, 1] memory blocks on x86 Gentoo Refleaks 3.6/3.x

STINNER Victor added the comment: I succeeded to reproduce a leak on the x86 Gentoo Refleaks 3.x buildbot using the following command. I'm unable to reproduce the bug on my Fedora 25. pydev@stormageddon ~/cpython $ ./python -m test -R 3:3 -j1 -m

[issue31217] test_code leaked [1, 1, 1] memory blocks on x86 Gentoo Refleaks 3.6/3.x

Changes by STINNER Victor : -- versions: +Python 3.7 ___ Python tracker ___ ___

[issue31217] test_code leaked [1, 1, 1] memory blocks on x86 Gentoo Refleaks 3.6/3.x

Changes by STINNER Victor : -- title: test_code leaked [1, 1, 1] memory blocks on x86 Gentoo Refleaks 3.6 -> test_code leaked [1, 1, 1] memory blocks on x86 Gentoo Refleaks 3.6/3.x ___ Python tracker