On Fri, Sep 8, 2017 at 11:36 AM, Neil Schemenauer < nas-python-id...@arctrix.com> wrote:
> On 2017-09-09, Chris Angelico wrote: > > Laziness has to be complete - or, looking the other way, eager > > importing is infectious. For foo to be lazy, bar also has to be lazy; > > Not with the approach I'm proposing. bar will be loaded in non-lazy > fashion at the right time, foo can still be lazy. > I'll bring the the conversation back here instead of co-opting the PEP 562 thread. On Sun, Sep 10, 2017 at 2:45 PM, Neil Schemenauer <n...@python.ca> wrote: > > I think the key is to make exec(code, module) work as an alternative > to exec(code, module.__dict). That allows module singleton classes > to define properties and use __getattr__. The > LOAD_NAME/STORE_NAME/DELETE_NAME opcodes need to be tweaked to > handle this. There will be a slight performance cost. Modules that > don't opt-in will not pay. > I'm not sure I follow the `exec(code, module)` part from the other thread. `exec` needs a dict to exec code into, the import protocol expects you to exec code into a module.__dict__, and even the related type.__prepare__ requires a dict so it can `exec` the class body there. Code wants a dict so functions created by the code string can bind it to function.__globals__. How do you handle lazy loading when a defined function requests a global via LOAD_NAME? Are you suggesting to change function.__globals__ to something not-a-dict, and/or change LOAD_NAME to bypass function.__globals__ and instead do something like: getattr(sys.modules[function.__globals__['__name__']], lazy_identifier) ? All this chatter about modifying opcodes, adding future statements, lazy module opt-in mechanisms, special handling of __init__ or __getattr__ or SOME_CONSTANT suggesting modules-are-almost-a-class-but-not-quite feel like an awful lot of work to me, adding even more cognitive load to an already massively complex import system. They seem to make modules even less like other objects or types. It would be really *really* nice if ModuleType got closer to being a simple class, instead of farther away. Maybe we start treating new modules like a subclass of ModuleType instead of all the half-way or special case solutions... HEAR ME OUT :-) Demo below. (also appended to end) https://gist.github.com/anthonyrisinger/b04f40a3611fd7cde10eed6bb68e8824 ``` # from os.path import realpath as rpath # from spam.ham import eggs, sausage as saus # print(rpath) # print(rpath('.')) # print(saus) $ python deferred_namespace.py <function realpath at 0x7f03db6b99d8> /home/anthony/devel/deferred_namespace Traceback (most recent call last): File "deferred_namespace.py", line 73, in <module> class ModuleType(metaclass=MetaModuleType): File "deferred_namespace.py", line 88, in ModuleType print(saus) File "deferred_namespace.py", line 48, in __missing__ resolved = deferred.__import__() File "deferred_namespace.py", line 9, in __import__ module = __import__(*self.args) ModuleNotFoundError: No module named 'spam' ``` Lazy-loading can be achieved by giving modules a __dict__ namespace that is import-aware. This parallels heavily with classes using __prepare__ to make their namespace order-aware (ignore the fact they are now order-aware by default). What if we brought the two closer together? I feel like the python object data model already has all the tools we need. The above uses __prepare__ and a module metaclass, but it could also use a custom __dict__ descriptor for ModuleType that returns an import-aware namespace (like DeferredImportNamespace in my gist). Or ModuleType.__new__ can reassign its own __dict__ (currently read-only). In all these cases we only need to make 2 small changes to Python: * Change `__import__` to call `globals.__defer__` (or similar) when appropriate instead of importing. * Create a way to make a non-binding class type so `module.function.__get__` doesn't create a bound method. The metaclass path also opens the door for passing keyword arguments to __prepare__ and __new__: from spam.ham import eggs using methods: True ... which might mean: GeneratedModuleClassName(ModuleType, methods=True): # module code ... # methods=True passed to __prepare__ and __new__, # allowing the module to implement bound methods! ... or even: import . import CustomMetaModule from spam.ham import ( eggs, sausage as saus, ) via CustomMetaModule using { methods: True, other: feature, } ... which might mean: GeneratedModuleClassName(ModuleType, metaclass=CustomMetaModule, methods=True, other=feature): # module code ... Making modules work like a real type/class means we we get __init__, __getattr__, and every other __*__ method *for free*, especially when combined with an extension to the import protocol allowing methods=True (or similar, like above). We could even subclass the namespace for each module, allowing us to effectively revert the module's __dict__ to a normal dict, and completely remove any possible overhead. Python types are powerful, let's do more of them! At the end of the day, I believe we should strive for these 3 things: * MUST work with function.__globals__[deferred], module.__dict__[deferred], and module.deferred. * SHOULD bring modules closer to normal objects, and maybe accept the fact they are more like class defe * SHOULD NOT require opt-in! Virtually every existing module will work fine. Thanks, ```python class Import: def __init__(self, args, attr): self.args = args self.attr = attr self.done = False def __import__(self): module = __import__(*self.args) if not self.attr: return module try: return getattr(module, self.attr) except AttributeError as e: raise ImportError(f'getattr({module!r}, {self.attr!r})') from e class DeferredImportNamespace(dict): def __init__(self, *args, **kwds): super().__init__(*args, **kwds) self.deferred = {} def __defer__(self, args, *names): # If __import__ is called and globals.__defer__() is defined, names to # bind are non-empty, each name is either missing from globals.deferred # or still marked done=False, then it should call: # # globals.__defer__(args, *names) # # where `args` are the original arguments and `names` are the bindings: # # from spam.ham import eggs, sausage as saus # __defer__(('spam.ham', self, self, ['eggs', 'sausage'], 0), 'eggs', 'saus') # # Records the import and what names would have been used. for i, name in enumerate(names): if name not in self.deferred: attr = args[3][i] if args[3] else None self.deferred[name] = Import(args, attr) def __missing__(self, name): # Raise KeyError if not a deferred import. deferred = self.deferred[name] try: # Replay original __import__ call. resolved = deferred.__import__() except KeyError as e: # KeyError -> ImportError so it's not swallowed by __missing__. raise ImportError(f'{name} = __import__{deferred.args}') from e else: # TODO: Still need a way to avoid binds... or maybe opt-in? # # from spam.ham import eggs, sausage using methods=True # # Save the import to namespace! self[name] = resolved finally: # Set after import to avoid recursion. deferred.done = True # Return import to original requestor. return resolved class MetaModuleType(type): @classmethod def __prepare__(cls, name, bases, defer=True, **kwds): return DeferredImportNamespace() if defer else {} class ModuleType(metaclass=MetaModuleType): # Simulate what we want to happen in a module block! __defer__ = locals().__defer__ # from os.path import realpath as rpath __defer__(('os.path', locals(), locals(), ['realpath'], 0), 'rpath') # from spam.ham import eggs, sausage as saus __defer__(('spam.ham', locals(), locals(), ['eggs', 'sausage'], 0), 'eggs', 'saus') # Good import. print(rpath) print(rpath('.')) # Bad import. print(saus) ``` -- C Anthony
_______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
