[issue22166] test_codecs leaks references

STINNER Victor added the comment: Codec tests don't leak anymore. I lost track of this very old issue. I don't think that anything should be done. Python code base changed a lot since 2014. I close the issue. -- resolution: -> fixed stage: needs patch -> resolved status: open ->

[issue22166] test_codecs leaks references

Stéphane Wirtel added the comment: For 3.7 and 3.8, I have just tried with this command to find leaks in the test_codec* but nothing :/ ./python -m test -v -l -R 6:6 -u all \ test_codecencodings_iso2022 \ test_codecs \ test_codecmaps_hk test_codecmaps_tw \

[issue22166] test_codecs leaks references

Change by Brett Cannon : -- nosy: -brett.cannon ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue22166] test_codecs leaks references

Change by Serhiy Storchaka : -- assignee: -> serhiy.storchaka ___ Python tracker ___ ___ Python-bugs-list mailing list

[issue22166] test_codecs leaks references

STINNER Victor added the comment: > Do you mind to create a PR Victor? I completely forgot this issue and I don't plan to fix it in the short-term. Feel free to write PR if you want. -- ___ Python tracker

[issue22166] test_codecs leaks references

Serhiy Storchaka added the comment: Do you mind to create a PR Victor? -- ___ Python tracker ___ ___ Python-bugs-list mailing list

[issue22166] test_codecs leaks references

Change by Serhiy Storchaka : -- versions: +Python 3.7, Python 3.8 -Python 3.4, Python 3.5 ___ Python tracker ___ ___

[issue22166] test_codecs leaks references

Nick Coghlan added the comment: Victor, would you be willing to take over pushing this one forward? I believe you have a much better idea of what's still needed than I do. -- assignee: ncoghlan - ___ Python tracker rep...@bugs.python.org

[issue22166] test_codecs leaks references

Changes by Serhiy Storchaka storch...@gmail.com: -- stage: resolved - needs patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22166 ___ ___

[issue22166] test_codecs leaks references

Changes by Nick Coghlan ncogh...@gmail.com: -- assignee: - ncoghlan ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22166 ___ ___ Python-bugs-list

[issue22166] test_codecs leaks references

Roundup Robot added the comment: New changeset fcf45ec7863e by Nick Coghlan in branch '3.4': Issue #22166: clear codec caches in test_codecs https://hg.python.org/cpython/rev/fcf45ec7863e New changeset 322ee2f2e922 by Nick Coghlan in branch 'default': Merge fix for issue #22166 from 3.4

[issue22166] test_codecs leaks references

Changes by Nick Coghlan ncogh...@gmail.com: -- resolution: - fixed stage: - resolved status: open - closed ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22166 ___

[issue22166] test_codecs leaks references

STINNER Victor added the comment: IMO test_codecs_fix1.patch is still needed. Review of Nick's change: +interp = PyThreadState_GET()-interp; +if (interp-codec_search_path == NULL) { +return -1; +} I don't think that this line raises an exception. You should use an

[issue22166] test_codecs leaks references

Marc-Andre Lemburg added the comment: On 15.09.2014 15:19, STINNER Victor wrote: STINNER Victor added the comment: IMO test_codecs_fix1.patch is still needed. Review of Nick's change: +interp = PyThreadState_GET()-interp; +if (interp-codec_search_path == NULL) { +

[issue22166] test_codecs leaks references

Nick Coghlan added the comment: Ah, I think I've found a possible underlying culprit: there's a separate C level codec_search_cache on the interpreter object that is independent of encodings._cache. The attached patch eliminates the reference leaks entirely (using gc.get_referrers() to get a

[issue22166] test_codecs leaks references

Nick Coghlan added the comment: Yep, it looks like the mess with the reference cycles in the tracebacks was just a secondary problem: the real culprit was the internal cache keeping everything else alive. Attached patch just adds clearing encodings._cache and the CPython internal cache to

[issue22166] test_codecs leaks references

Antoine Pitrou added the comment: I'd much rather see a private function somewhere (in the _codecs module) to clear that private interpreter cache. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22166

[issue22166] test_codecs leaks references

Nick Coghlan added the comment: Yeah, I'm wary of the GC hack as well. It's a nice way to prove that this the internal cache is the problem, but probably not something we want to commit. A codecs._forget_codec helper in the C module would be all that was needed to handle it. Other

[issue22166] test_codecs leaks references

Marc-Andre Lemburg added the comment: On 06.09.2014 15:03, Nick Coghlan wrote: A codecs._forget_codec helper in the C module would be all that was needed to handle it. Other implementations could then also use that to clear their own internal cache (if they have one). If you want to take

[issue22166] test_codecs leaks references

Serhiy Storchaka added the comment: May be just do not use the unique name? -- nosy: +serhiy.storchaka ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22166 ___

[issue22166] test_codecs leaks references

Nick Coghlan added the comment: This investigation also showed why the original tests that used a non-unique name didn't work: the codec was getting cached the first time through and hence not seeing the different codecs registered by other tests. MAL, my plan was to provide an API to forget a

[issue22166] test_codecs leaks references

Marc-Andre Lemburg added the comment: On 06.09.2014 21:14, Nick Coghlan wrote: This investigation also showed why the original tests that used a non-unique name didn't work: the codec was getting cached the first time through and hence not seeing the different codecs registered by other

[issue22166] test_codecs leaks references

Antoine Pitrou added the comment: Well, the point here is to add a private API just for our test suite, not something users would want to call. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22166

[issue22166] test_codecs leaks references

Nick Coghlan added the comment: Targeted API to forget a particular codec name. Normally we don't care about the caching (as evidenced by the fact it took 14+ years to want this feature). The intent with this API is to be able to selective purge a particular codec when we either care about

[issue22166] test_codecs leaks references

Nick Coghlan added the comment: Yeah, the unique name was a hack to deal with the fact I couldn't unregister anything. As a simpler short term fix for the reference leak, perhaps it would be worth trying a call to traceback.clear_frames(caught) as a new final line in

[issue22166] test_codecs leaks references

STINNER Victor added the comment: Attached: Draft patch to fix some reference leaks in test_codecs. -- title: test_codecs leaking references - test_codecs leaks references Added file: http://bugs.python.org/file36524/test_codecs_fix1.patch ___ Python

[issue22166] test_codecs leaks references

STINNER Victor added the comment: ExceptionChainingTest creates a random codec name. If you change the codec name to a fixed string like xxx, ExceptionChainingTest.test_raise_by_type() doesn't leak anymore (when test_codecs_fix1.patch is applied), but other tests start to fail. We should

[issue22166] test_codecs leaks references

Antoine Pitrou added the comment: We should maybe enhance the codecs API to be able to unregister a codec? That would be nice. At least as a private API in 3.4. -- nosy: +pitrou ___ Python tracker rep...@bugs.python.org