[issue17621] Create a lazy import loader mixin
Roundup Robot added the comment: New changeset 52b58618199c by Brett Cannon in branch 'default': Issue #17621: Introduce importlib.util.LazyLoader. http://hg.python.org/cpython/rev/52b58618199c -- nosy: +python-dev ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue17621 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue17621] Create a lazy import loader mixin
Brett Cannon added the comment: I went ahead and committed. I realized I could loosen the no create_module() requirement, but I think it could lead to more trouble than it's worth so I left it as-is for now. If people says it's an issue we can revisit it. -- resolution: - fixed stage: patch review - committed/rejected status: open - closed ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue17621 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue17621] Create a lazy import loader mixin
Eric Snow added the comment: Sweet! -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue17621 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue17621] Create a lazy import loader mixin
Brett Cannon added the comment: So as-is, this won't help with startup as we already make sure that no unnecessary modules are loaded during startup. But one way we do that is through local imports in key modules, e.g. os.get_exec_path(). So what we could consider is instead of doing a local import we use lazy imports. We could introduce importlib._lazy_import() which could be (roughly): def _lazy_import(fullname): try: return sys.modules[fullname] except KeyError: spec = find_spec(fullname) loader = LazyLoader(spec.loader) # Make module with proper locking and get it inserted into sys.modules. loader.exec_module(module) return module I don't know if that simplifies things, though, compared to a local import. It might help once a module is identified as on the critical path of startup since all global imports in that module could be lazy, but we would still need to identify those critical modules. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue17621 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue17621] Create a lazy import loader mixin
Changes by Thomas Wouters tho...@python.org: -- nosy: +twouters ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue17621 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue17621] Create a lazy import loader mixin
Changes by Augie Fackler li...@durin42.com: -- nosy: +durin42 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue17621 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue17621] Create a lazy import loader mixin
Eric Snow added the comment: I wonder if there would be any benefit to using this for some of the modules that get loaded during startup. I seem to remember there being a few for which lazy loading would have an effect. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue17621 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue17621] Create a lazy import loader mixin
Eric Snow added the comment: New review posted. Basically LGTM. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue17621 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue17621] Create a lazy import loader mixin
Brett Cannon added the comment: Another update to trigger loading on attribute deletions as well as detecting when an import swapped the module in sys.modules, raising ValueError if detected since it won't have the affect that's expected (could be convinced to make that ImportError instead). -- Added file: http://bugs.python.org/file34591/lazy_loader.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue17621 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue17621] Create a lazy import loader mixin
Eric Snow added the comment: Review posted. Thanks for working on this, Brett. -- nosy: +eric.snow ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue17621 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue17621] Create a lazy import loader mixin
Changes by Jakub Wilk jw...@jwilk.net: -- nosy: +jwilk ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue17621 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue17621] Create a lazy import loader mixin
Brett Cannon added the comment: Here is a new patch that addresses Eric's comments and also fills in some holes that I realized I had while fixing things up. PTAL. -- Added file: http://bugs.python.org/file34575/lazy_loader.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue17621 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue17621] Create a lazy import loader mixin
Changes by Brett Cannon br...@python.org: Removed file: http://bugs.python.org/file34575/lazy_loader.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue17621 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue17621] Create a lazy import loader mixin
Changes by Brett Cannon br...@python.org: Added file: http://bugs.python.org/file34576/lazy_loader.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue17621 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue17621] Create a lazy import loader mixin
Brett Cannon added the comment: New patch that includes docs and integrates the tests. If someone who understands import can look it over and give me an LGTM that would be appreciated. -- Added file: http://bugs.python.org/file34556/lazy_loader.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue17621 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue17621] Create a lazy import loader mixin
Brett Cannon added the comment: Here is a patch which implements a lazy loader for loaders that define exec_module(). -- keywords: +patch Added file: http://bugs.python.org/file33947/lazy_loader.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue17621 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue17621] Create a lazy import loader mixin
Changes by Brett Cannon br...@python.org: -- dependencies: -Make isinstance() work with super type instances ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue17621 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue17621] Create a lazy import loader mixin
Changes by Brett Cannon br...@python.org: -- stage: test needed - patch review ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue17621 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue17621] Create a lazy import loader mixin
Brett Cannon added the comment: Need to quickly test that this will work with PEP 451 works with the design in my head before we get farther into 3.4. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue17621 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue17621] Create a lazy import loader mixin
Brett Cannon added the comment: Attached is a script to verify that PEP 451 works as desired, so Python 3.5 doesn't have any technical blockers for doing a lazy loader for PEP 451 loaders. And with __spec__.loader_state it might be possible to work things out through a common API to work around issue #18275 so that relying on super() and doing this as a mixin goes away and instead just somehow store the final loader on the spec (e.g. loader's constructor just takes a spec so that you can instantiate the actual loader, reset __loader__ __spec__.loader, and then proceed with exec_module()). -- dependencies: -Implementation for PEP 451 (importlib.machinery.ModuleSpec) Added file: http://bugs.python.org/file33136/lazy_test.py ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue17621 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue17621] Create a lazy import loader mixin
Brett Cannon added the comment: Shifting this to Python 3.5 to rework using the forthcoming PEP 451 APIs which have been designed to explicitly allow for a lazy loader. -- versions: +Python 3.5 -Python 3.4 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue17621 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue17621] Create a lazy import loader mixin
Brett Cannon added the comment: Won't work until PEP 451 code lands and lets us drop __package__/__loader__ checking/setting post-load. -- dependencies: +Implementation for PEP 451 (importlib.machinery.ModuleSpec) ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue17621 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue17621] Create a lazy import loader mixin
Brett Cannon added the comment: Import manages the lock, not loaders. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue17621 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue17621] Create a lazy import loader mixin
Richard Oudkerk added the comment: I was thinking about the line self.__dict__.update(state) overwriting new data with stale data. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue17621 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue17621] Create a lazy import loader mixin
Brett Cannon added the comment: It still falls under the purview of import to manage that lock. It's just the design of the API from PEP 302. Otherwise it's just like any other reload. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue17621 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue17621] Create a lazy import loader mixin
Changes by Arfrever Frehtes Taifersar Arahesis arfrever@gmail.com: -- nosy: +Arfrever ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue17621 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue17621] Create a lazy import loader mixin
Richard Oudkerk added the comment: Apologies for being dense, but how would you actually use such a loader? Would you need to install something in sys.meta_path/sys.path_hooks? Would it make all imports lazy or only imports of specified modules? -- nosy: +sbt ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue17621 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue17621] Create a lazy import loader mixin
Brett Cannon added the comment: So the approaches I have been using make a loader lazy, so what you have to change in terms of sys.meta_path, sys.path_hooks, etc. would very from loader to loader. I have realized one tricky thing with all of this is that importlib itself inspects modules post-import to verify that __loader__ and __package__ have been set. That typically triggers an immediate load and so might need to be special-cased. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue17621 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue17621] Create a lazy import loader mixin
Changes by Brett Cannon br...@python.org: Added file: http://bugs.python.org/file30670/test_lazy_loader.py ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue17621 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue17621] Create a lazy import loader mixin
Changes by Brett Cannon br...@python.org: Added file: http://bugs.python.org/file30671/lazy_mixin.py ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue17621 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue17621] Create a lazy import loader mixin
Brett Cannon added the comment: I have attached the test suite and two versions: one using a mixin and one using a proxy. Neither solve the issue of import touching the module in any way to check __loader__ and __package__. The mixin version is also failing one test which could quite possibly be a pain to fix and so it might cause me to prefer the proxy solution. -- Added file: http://bugs.python.org/file30672/lazy_proxy.py ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue17621 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue17621] Create a lazy import loader mixin
Richard Oudkerk added the comment: Shouldn't the import lock be held to make it threadsafe? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue17621 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue17621] Create a lazy import loader mixin
Brett Cannon added the comment: While seing if it was worth making isinstance work with super(), I came up with this at Antoine's suggestion of using a proxy instead of a mixin: class LazyProxy(importlib.abc.Loader): def __init__(self, loader): self.loader = loader def __call__(self, *args, **kwargs): self.args = args self.kwargs = kwargs return self def load_module(self, fullname): # XXX ignoring sys.modules details, e.g. if module already existed. lazy_module = LazyModule(fullname, proxy=self, name=fullname) sys.modules[fullname] = lazy_module return lazy_module class LazyModule(types.ModuleType): def __init__(*args, proxy, name, **kwargs): self.__proxy = proxy self.__name = name super().__init__(*args, **kwargs) def __getattribute__(self, attr): self.__class__ = Module state = self.__dict__.copy() loader = self.__proxy.loader(*self.proxy.args, **self.proxy.kwargs) # XXX ignoring sys.modules details, e.g. removing module on load failure. loader.load_module(self.__name) self.__dict__.update(state) return getattr(module, attr) -- dependencies: +Make isinstance() work with super type instances ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue17621 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue17621] Create a lazy import loader mixin
Brett Cannon added the comment: I think the first step for this bug, now that I have two possible approaches, is to write the unit tests. That way both approaches can equally be validated based on their merits of complexity, etc. while verifying the work properly. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue17621 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue17621] Create a lazy import loader mixin
Brett Cannon added the comment: One problem with the importers code is it doesn't properly clean up after the module if the load failed and it wasn't a reload. Should store an attribute on the module signaling whether it was a reload or not to know whether an exception raised during loading should lead to the module being removed from sys.modules. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue17621 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue17621] Create a lazy import loader mixin
New submission from Brett Cannon: People keep asking and I keep promising to get a lazy loader into Python 3.4. This issue is for me to track what I have done towards meeting that promise. To start, I have something working at https://code.google.com/p/importers/, but I need to make sure that the code copies any newly assigned objects post-import but before completing an attribute read:: import lazy_mod lazy_mod.attr = True # Does not have to trigger import, although there is nothing wrong if it did. lazy_mod.func() # Since it might depend on 'attr', don't return attr until after import and 'attr' with a value of True has been set. Also need to see if anything can be done about isinstance/issubclass checks as super() is used for assigning to __loader__ and thus might break checks for ABCs. Maybe create a class from scratch w/o the mixin somehow (which I don't see from looking at http://docs.python.org/3.4/library/types.html#module-types w/o re-initializing everything)? Somehow get __subclasscheck__() on the super() class? Some other crazy solution that avoids having to call __init__() a second time? -- assignee: brett.cannon components: Library (Lib) messages: 185852 nosy: brett.cannon priority: low severity: normal stage: test needed status: open title: Create a lazy import loader mixin type: enhancement versions: Python 3.4 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue17621 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue17621] Create a lazy import loader mixin
Changes by Barry A. Warsaw ba...@python.org: -- nosy: +barry ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue17621 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue17621] Create a lazy import loader mixin
Changes by Eric V. Smith e...@trueblade.com: -- nosy: +eric.smith ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue17621 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
