[issue43658] implementations of the deprecated load_module import loader API, as prescribed by the documentation, are not thread safe
New submission from kale-smoothie : Unless I've misread or misunderstood, the documentation at https://docs.python.org/3/reference/import.html#loaders for the deprecated `load_module` method doesn't indicate any requirements or caveats for thread safe importing. As it stands, I think it is not thread-safe, since the module is not protected against concurrent imports by the internal implementation marker `__spec__._initializing = True`. Additionally, the deprecated function decorator, `importlib.util.module_for_loader` seems to implement the marker incorrectly (sets `__initializing__` directly on the module). I think this behaviour should either be documented as a major caveat, or internal details exposed to allow thread-safe implementations, or the old API removed entirely. -- assignee: docs@python components: Documentation files: thread_unsafe_import.py messages: 389713 nosy: docs@python, kale-smoothie priority: normal severity: normal status: open title: implementations of the deprecated load_module import loader API, as prescribed by the documentation, are not thread safe type: behavior versions: Python 3.10, Python 3.6, Python 3.7, Python 3.8, Python 3.9 Added file: https://bugs.python.org/file49916/thread_unsafe_import.py ___ Python tracker <https://bugs.python.org/issue43658> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue41422] C Unpickler memory leak via memo
kale-smoothie added the comment: The leak demonstrated in the attachment is, to my understanding, caused by memoizing the closure returned from the `find_class` method that's used to intercept global references. The cycle is then: Unpickler, memo table, closure, Unpickler (via cell reference to `self`). My proposed patch visits every entry in the memo table. Pre-patch run of valgrind on leak_pickler.py: ==20339== HEAP SUMMARY: ==20339== in use at exit: 190,189,238 bytes in 2,406,919 blocks ==20339== total heap usage: 3,150,288 allocs, 743,369 frees, 233,766,596 bytes allocated ==20339== ==20339== LEAK SUMMARY: ==20339==definitely lost: 0 bytes in 0 blocks ==20339==indirectly lost: 0 bytes in 0 blocks ==20339== possibly lost: 190,176,150 bytes in 2,406,835 blocks ==20339==still reachable: 13,088 bytes in 84 blocks ==20339== suppressed: 0 bytes in 0 blocks ==20339== Rerun with --leak-check=full to see details of leaked memory Post-patch run of valgrind on leak_pickler.py: ==20880== HEAP SUMMARY: ==20880== in use at exit: 667,277 bytes in 6,725 blocks ==20880== total heap usage: 2,853,739 allocs, 2,847,014 frees, 216,473,216 bytes allocated ==20880== ==20880== LEAK SUMMARY: ==20880==definitely lost: 0 bytes in 0 blocks ==20880==indirectly lost: 0 bytes in 0 blocks ==20880== possibly lost: 654,624 bytes in 6,646 blocks ==20880==still reachable: 12,653 bytes in 79 blocks ==20880== suppressed: 0 bytes in 0 blocks ==20880== Rerun with --leak-check=full to see details of leaked memory -- ___ Python tracker <https://bugs.python.org/issue41422> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue41422] C Unpickler memory leak via memo
New submission from kale-smoothie : I'm not familiar with the workings of GC/pickle, but it looks like the traverse code in the C Unpickler omits a visit to the memo, potentially causing a memory leak? -- components: Library (Lib) files: leak_pickler.py messages: 374518 nosy: kale-smoothie priority: normal severity: normal status: open title: C Unpickler memory leak via memo type: resource usage versions: Python 3.10, Python 3.5, Python 3.6, Python 3.7, Python 3.8, Python 3.9 Added file: https://bugs.python.org/file49345/leak_pickler.py ___ Python tracker <https://bugs.python.org/issue41422> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
