[issue46963] Deep Lazy Imports - Interpreter-level deferred module loading
New submission from Germán Méndez Bravo : As the size of a Python project increases, the number of modules and the complexity of its dependencies increases too, producing two problems in large codebases: increased risk of import cycles and slow start times due to the number of modules that tend to need getting loaded. We propose implementing a robust and transparent lazy loader in CPython, based on an existing implementation we have in Cinder that proved to be extremely valuable to thousands of developers across Meta, and battle-tested in Instagram Server in production (https://docs.google.com/document/d/1l8I-FDE1xrIShm9eSNJqsGmY_VanMDX5-aK_gujhYBI/edit#heading=h.pu7ja6wu0ib). Our internal implementation is based on introducing deferred objects and converting top-level imported names to these deferred objects that are evaluated on first use. Enabling this across multiple types of workloads in Meta consistently shown improvements in startup times (up to 70%) and memory footprint (up to 40%), while virtually eliminating occurrences of import cycles. Converting imported names to deferred objects is a semantic change, and the benefits are usually meaningful only at very large scale, so we propose not changing the default behavior, and making it possible for users to opt-in to this lazy loading mechanism via -x flags or environment variables. This change would require related documentation changes that would cover the gotchas and edge cases, such as: some packages rely and import-time side-effects for correct operation; nested modules can’t be accessed unless imported explicitly; deferred loading may also defer exceptions from import time and first-access time which could be confusing. The import-time side effects gotcha can be mitigated using eager-import lists and directives, though it could be desirable to use this as a forcing function to discourage package maintainers from relying on these side-effects. -- components: Interpreter Core messages: 414770 nosy: Kronuz, carljm, dino.viehland, itamaro priority: normal severity: normal status: open title: Deep Lazy Imports - Interpreter-level deferred module loading type: enhancement versions: Python 3.11 ___ Python tracker <https://bugs.python.org/issue46963> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue41249] TypedDict inheritance doesn't work with get_type_hints and postponed evaluation of annotations across modules
Germán Méndez Bravo added the comment: I added a pull request with my fix here: https://github.com/python/cpython/pull/27017 -- ___ Python tracker <https://bugs.python.org/issue41249> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue41249] TypedDict inheritance doesn't work with get_type_hints and postponed evaluation of annotations across modules
Change by Germán Méndez Bravo : -- keywords: +patch pull_requests: +25576 stage: -> patch review pull_request: https://github.com/python/cpython/pull/27017 ___ Python tracker <https://bugs.python.org/issue41249> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue41249] TypedDict inheritance doesn't work with get_type_hints and postponed evaluation of annotations across modules
Germán Méndez Bravo added the comment: Nils, unfortunately, fixing the MRO here won’t fix the issue because `TypedDict.__annotations__` in the class copies the annotations from the parent classes, and when the type evaluation is made, it’s made using the copied annotation found in the bottommost class (which is thus then expected to be a forward reference in the same module as the class that inherited them. This producing the exact same problem of missing type. The most likely reason for incomplete MRO is that `TypeDict` extends a class’ `__annotations__` with all of it’s parent’s `__annotations__`, so the final class has the complete set of annotations of all of its parents, so having the full inheritance chain made less sense, after all the final dictionary class has all the annotations. > On Jul 3, 2021, at 13:43, Nils Kattenbeck wrote: > > > Nils Kattenbeck added the comment: > >> The way I fixed this is I added `__forward_module__` to `typing.ForwardRef`, >> so that it can resolve the forward reference with the same globals as the >> ones specified by the module in `__forward_module__`. `TypedDict`'s >> metaclass should then pass the dictionary’s module name to the annotations’ >> forward references via the added `module`’s keyword argument in >> `typing._type_check()`. I can work in a pull request with this solution and >> discuss any potential problems. > > While this seems like a good solution I would still like to figure out why > TypedDict do not preserve MRO. Because for now I have not found a reason nor > did someone on the typing-sig mailinglist have a clue. Should there (no > longer) be a reason for this then this problem has a trivial solution (just > re-add the MRO and use that). > > -- > > ___ > Python tracker > <https://bugs.python.org/issue41249> > ___ -- ___ Python tracker <https://bugs.python.org/issue41249> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue41249] TypedDict inheritance doesn't work with get_type_hints and postponed evaluation of annotations across modules
Germán Méndez Bravo added the comment: The way I fixed this is I added `__forward_module__` to `typing.ForwardRef`, so that it can resolve the forward reference with the same globals as the ones specified by the module in `__forward_module__`. `TypedDict`'s metaclass should then pass the dictionary’s module name to the annotations’ forward references via the added `module`’s keyword argument in `typing._type_check()`. I can work in a pull request with this solution and discuss any potential problems. -- nosy: +Kronuz ___ Python tracker <https://bugs.python.org/issue41249> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue43392] Optimize repeated calls to `__import__()`
Change by Germán Méndez Bravo : -- keywords: +patch pull_requests: +23506 stage: -> patch review pull_request: https://github.com/python/cpython/pull/24735 ___ Python tracker <https://bugs.python.org/issue43392> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue43392] Optimize repeated calls to `__import__()`
New submission from Germán Méndez Bravo : A call to `importlib.__import__()` normally locks the import for the module being worked on; this, however has a performance impact for modules that are already imported and fully initialized. An example of this are inline `__import__()` calls in a hot path that is called repeatedly during the life of the process. Proposal: A two steps check in `importlib._bootstrap._find_and_load()` to avoid locking when the module has been already imported and it's ready. -- components: Library (Lib) messages: 388057 nosy: Kronuz priority: normal severity: normal status: open title: Optimize repeated calls to `__import__()` versions: Python 3.10, Python 3.6, Python 3.7, Python 3.8, Python 3.9 ___ Python tracker <https://bugs.python.org/issue43392> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
