[issue27535] Ignored ResourceWarning warnings leak memory in warnings registries

STINNER Victor added the comment: Alexander Mohr reported a memory leak in his code using botocore (Python client for Amazon S3): bpo-33565. His code emited ResourceWarning, but these warnings are ignored by default. The link between ignored warnings and a memory leak is

[issue27535] Ignored ResourceWarning warnings leak memory in warnings registries

Alexander Mohr added the comment: not fixing this means that 3.6 slowly leaks for many people in prod. It's not often possible to fix all the warnings on large dynamic applications, I highly suggest finding a way to get this into 3.6. I bet there are a lot of

[issue27535] Ignored ResourceWarning warnings leak memory in warnings registries

STINNER Victor added the comment: I ran a quick benchmark on Python 3.6: current code (3.6 branch, ref) => PR 4587 (patch): vstinner@apu$ python3 -m perf compare_to ref.json patch.json Mean +- std dev: [ref] 597 ns +- 10 ns -> [patch] 830 ns +- 15 ns: 1.39x slower

[issue27535] Ignored ResourceWarning warnings leak memory in warnings registries

STINNER Victor added the comment: Serhiy: "I meant the original issue. AFAIK ResourceWarningis not raised in 2.7. And other warnings less likely have unique messages. Seems the problem is less critical in 2.7." Oh yes, sorry, I forgot your comment and I forgot that

[issue27535] Ignored ResourceWarning warnings leak memory in warnings registries

Serhiy Storchaka added the comment: I meant the original issue. AFAIK ResourceWarningis not raised in 2.7. And other warnings less likely have unique messages. Seems the problem is less critical in 2.7. -- ___ Python

[issue27535] Ignored ResourceWarning warnings leak memory in warnings registries

STINNER Victor added the comment: Serhiy: "As for 2.7, does this problem exist in 2.7?" Yes, see my PR 4588. If you run the test without the fix, the test fails. -- ___ Python tracker

[issue27535] Ignored ResourceWarning warnings leak memory in warnings registries

Serhiy Storchaka added the comment: I'm not sure. Ask the RM for 3.6. As for 2.7, does this problem exist in 2.7? -- ___ Python tracker

[issue27535] Ignored ResourceWarning warnings leak memory in warnings registries

STINNER Victor added the comment: Victor: "I consider that PR 4489 is a bugfix, so I proposed backports to Python 2.7 (PR 4588) and 3.6 (PR 4587)." Serhiy: Are you ok to backport the change to stable branches? -- ___

[issue27535] Ignored ResourceWarning warnings leak memory in warnings registries

Change by STINNER Victor : -- pull_requests: +4511 ___ Python tracker ___ ___

[issue27535] Ignored ResourceWarning warnings leak memory in warnings registries

STINNER Victor added the comment: I consider that PR 4489 is a bugfix, so I proposed backports to Python 2.7 (PR 4588) and 3.6 (PR 4587). -- ___ Python tracker

[issue27535] Ignored ResourceWarning warnings leak memory in warnings registries

Serhiy Storchaka added the comment: Thank you Victor! -- assignee: serhiy.storchaka -> vstinner versions: +Python 3.7 -Python 3.6 ___ Python tracker

[issue27535] Ignored ResourceWarning warnings leak memory in warnings registries

Change by Roundup Robot : -- pull_requests: +4510 ___ Python tracker ___

[issue27535] Ignored ResourceWarning warnings leak memory in warnings registries

STINNER Victor added the comment: https://github.com/python/cpython/pull/4489#issuecomment-346673704 Serhiy: "Could you please make a benchmark for warnings emitted in a tight loop in the C code. It would be nice to know what is the worst case. If it is less than 2x

[issue27535] Ignored ResourceWarning warnings leak memory in warnings registries

STINNER Victor added the comment: New changeset c9758784eb321fb9771e0bc7205b296e4d658045 by Victor Stinner in branch 'master': bpo-27535: Fix memory leak with warnings ignore (#4489) https://github.com/python/cpython/commit/c9758784eb321fb9771e0bc7205b296e4d658045

[issue27535] Ignored ResourceWarning warnings leak memory in warnings registries

STINNER Victor added the comment: > These results look strange to me. I expected the same difference but smaller > the base time. Honestly, I was also surprised. I checked the whole benchmark twice. I also rebased the PR locally to make sure that it's not a side

[issue27535] Ignored ResourceWarning warnings leak memory in warnings registries

Serhiy Storchaka added the comment: These results look strange to me. I expected the same difference but smaller the base time. -- ___ Python tracker

[issue27535] Ignored ResourceWarning warnings leak memory in warnings registries

Change by STINNER Victor : Added file: https://bugs.python.org/file47287/bench_ignore_warn_c.py ___ Python tracker ___

[issue27535] Ignored ResourceWarning warnings leak memory in warnings registries

STINNER Victor added the comment: New benchmark on the emitting a warning which is ignored. Benchmark the PR 4489. Warning emitted in Python, warnings.warn(): vstinner@apu$ ./python -m perf compare_to master.json ignore.json Mean +- std dev: [master] 705 ns +- 24

[issue27535] Ignored ResourceWarning warnings leak memory in warnings registries

STINNER Victor added the comment: New changeset b98f1715a35d2cbd1e9e45e1e7ae51a39e00dc4a by Victor Stinner in branch 'master': bpo-27535: Cleanup create_filter() (#4516) https://github.com/python/cpython/commit/b98f1715a35d2cbd1e9e45e1e7ae51a39e00dc4a --

[issue27535] Ignored ResourceWarning warnings leak memory in warnings registries

Change by STINNER Victor : -- pull_requests: +4453 ___ Python tracker ___ ___

[issue27535] Ignored ResourceWarning warnings leak memory in warnings registries

STINNER Victor added the comment: My PR 4502 implements all optimizations, but also converts warnings.filters to a tuple, to detect when detect attempts to modify directly warnings.filters, rather than using function calls. Problem: test.libregrtest module uses a

[issue27535] Ignored ResourceWarning warnings leak memory in warnings registries

STINNER Victor added the comment: New changeset 82656276caf4cb889193572d2d14dbc5f3d2bdff by Victor Stinner in branch 'master': bpo-27535: Optimize warnings.warn() (#4508) https://github.com/python/cpython/commit/82656276caf4cb889193572d2d14dbc5f3d2bdff --

[issue27535] Ignored ResourceWarning warnings leak memory in warnings registries

Change by STINNER Victor : -- pull_requests: +4445 ___ Python tracker ___ ___

[issue27535] Ignored ResourceWarning warnings leak memory in warnings registries

STINNER Victor added the comment: I published my WIP work on optimizing warnings: PR 4502, it's based on PR 4489. The code should be cleaned up, especially the "_PyRuntime" part. The default action caching should be fixed. --

[issue27535] Ignored ResourceWarning warnings leak memory in warnings registries

Change by STINNER Victor : -- pull_requests: +4440 ___ Python tracker ___ ___

[issue27535] Ignored ResourceWarning warnings leak memory in warnings registries

STINNER Victor added the comment: I implemented a cache for warnings.filters in C. With my WIP patch which doesn't touch the registry for the ignore action, warnings.warn() is now faster than the current code ;-) haypo@selma$ ./python -m perf compare_to ref.json

[issue27535] Ignored ResourceWarning warnings leak memory in warnings registries

STINNER Victor added the comment: The performance bottleneck of warnings.warn() is the dance between the C _warnings module and the Python warnings module. The C code retrieves many attributes, like warnings.filters, at each call, and does conversion from Python

[issue27535] Ignored ResourceWarning warnings leak memory in warnings registries

STINNER Victor added the comment: > $ ./python -m perf compare_to ref.json patch.json > Mean +- std dev: [ref] 903 ns +- 70 ns -> [patch] 1.06 us +- 0.06 us: 1.17x > slower (+17%) We are talking about a difference of 157 nanoseconds. On the same laptop, a Python

[issue27535] Ignored ResourceWarning warnings leak memory in warnings registries

Serhiy Storchaka added the comment: I think it is common if filters don't depend on the warning test. In this case we can test and set the key (None, category, lineno) before (text, category, lineno). -- ___ Python

[issue27535] Ignored ResourceWarning warnings leak memory in warnings registries

Nick Coghlan added the comment: Touching the filesystem implicitly can have all sorts of unintentional side effects (especially in read-only environments), so we strongly prefer not to do that. The most general solution here would be to turn the warnings registry into an

[issue27535] Ignored ResourceWarning warnings leak memory in warnings registries

Decorater added the comment: If it was me I would store the warning registry in an error log or something in the current directory that ran python itself. (maybe something like ``[main script name].log``? This way it generates the warnings like usual and does not eat

[issue27535] Ignored ResourceWarning warnings leak memory in warnings registries

STINNER Victor added the comment: Attached bench_ignore_warn.py: microbenchmark (using the perf module) to measure the performance of emitting a warning when the warning is ignored. I added two basic optimizations to my PR 4489. With these optimizations, the

[issue27535] Ignored ResourceWarning warnings leak memory in warnings registries

Serhiy Storchaka added the comment: > While I don't see how we can avoid "leaking memory" (growing the registry) > for actions like "once", I think that it's ok to don't touch the registry for > the "ignore" action. So at least, an application ignoring

[issue27535] Ignored ResourceWarning warnings leak memory in warnings registries

Jonathan Bastien-Filiatrault added the comment: > But it will still "leak" when you display ResourceWarning warnings with an > action different than "always". In this case, IMHO the root issue is more the > code which doesn't close the resource, than Python itself. Not

[issue27535] Ignored ResourceWarning warnings leak memory in warnings registries

STINNER Victor added the comment: > The best way to fix this is excluding file name from warning message. As a > result, the message for every file will be the same and warnings registry > will not grow. Your warning comes from the io module which uses the message:

[issue27535] Ignored ResourceWarning warnings leak memory in warnings registries

Change by STINNER Victor : -- title: Memory leaks when opening tons of files -> Ignored ResourceWarning warnings leak memory in warnings registries ___ Python tracker