Re: [Python-Dev] PEP 382 progress: import hooks
On Fri, Jul 23, 2010 at 09:54, P.J. Eby p...@telecommunity.com wrote: At 11:57 AM 7/23/2010 +0100, Brett Cannon wrote: On Thu, Jul 22, 2010 at 19:19, P.J. Eby mailto:p...@telecommunity.com p...@telecommunity.com wrote: What does is not a package actually mean in that context? The module is a module but not a package. Um... that's not any clearer. Are you saying that a module of the same name takes precedence over a package? Is that the current precedence as well? No, packages take precedence. I meant that something is a module but it is not a package; a package implicitly includes a module, but a module is not automatically a package. Regarding load_module_with_path(), how does its specification differ from simply creating a module in sys.modules, setting its __path__, and then invoking the standard load_module()? Ā (i.e., is this method actually needed, since a correct PEP 302 loader *must* reuse an existing module object in sys.modules) It must reuse the module itself but a proper reload would reset __path__ as leaving it unchanged is not a proper resetting of the module object. So this module is needed in order to force the loader Um, no. Reloading doesn't reset the module contents, not even __path__. Never has, from Python 2.2 through 2.7 -- even in 3.1. At least, not for normal filesystem .py/.pyc files. (I tested with 'os', adding an extra 'foo' attribute, and also setting a __path__; both were unaffected by reload(), in all 7 Python versions. Perhaps you're saying this happens with zipfiles, or packages that already have a __path__, or...? It's how I implemented it in importlib and it passes Python's unit test suite that way; zipimport also does it this way as it too does not differentiate a reload from a clean load beyond grabbing the module from sys.modules if it is already there. PEP 302 does not directly state that reloading should not reset the attributes that import must set, simply that a module from sys.modules must be reused. Since zipimport does it this way I wouldn't count on other loaders not setting __path__. Am I correct in understanding that, as written, one would have to redefine __import__ to implement this in a library for older Python versions? Ā Or is it implementable as a meta_path importer? Redefine __import__ (unless Martin and I are missing something, but I tried to think of how to implement this use sys.meta_path and couldn't come up with a solution). I'm thinking it *could* be done with a meta_path hook, but only by doubling the search length in the event that the search failed. That seems a bit icky, but replacing the entire import process seems ickier (more code surface to maintain, more bug potential) in the case of supporting older Pythons. I'm personally not worried about supporting older versions of Python as this is a new feature. Better to design it properly than come up with some hack solution as we will all have to live with this for a long time. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 382 progress: import hooks
At 05:28 PM 8/2/2010 -0700, Brett Cannon wrote: On Fri, Jul 23, 2010 at 09:54, P.J. Eby mailto:p...@telecommunity.comp...@telecommunity.com wrote: At 11:57 AM 7/23/2010 +0100, Brett Cannon wrote: On Thu, Jul 22, 2010 at 19:19, P.J. Eby mailto:p...@telecommunity.comp...@telecommunity.com wrote: What does is not a package actually mean in that context? The module is a module but not a package. Um... Â that's not any clearer. Â Are you saying that a module of the same name takes precedence over a package? Â Is that the current precedence as well? No, packages take precedence. I meant that something is a module but it is not a package; a package implicitly includes a module, but a module is not automatically a package. That explanation still isn't making things any clearer for me. That is, I don't know how to get from that statement to actual code, even if I were writing a filesystem or zip importer, let alone anything more exotic. zipimport also does it this way as it too does not differentiate a reload from a clean load beyond grabbing the module from sys.modules if it is already there. PEP 302 does not directly state that reloading should not reset the attributes that import must set, simply that a module from sys.modules must be reused. Since zipimport does it this way I wouldn't count on other loaders not setting __path__. Fair enough, though certainly unfortunate. In particular, it means that it's not actually possible to correctly/completely implement PEP 382 on any already-released version of Python, without essentially replacing zipimport. (Unless the spec can be tweaked a bit.) I'm personally not worried about supporting older versions of Python as this is a new feature. Better to design it properly than come up with some hack solution as we will all have to live with this for a long time. Currently, older Pythons are the only versions I *do* support, so I'm very concerned with it. Otherwise, I'd not be asking all these questions. ;-) Personally, I think there are features in the PEP that make things unnecessarily complicated - for example, supporting both __init__.py *and* .pth files in the same directory. If it were either/or, it would be a LOT easier to implement on older Pythons, since it wouldn't matter when you initialized the __path__ in that case. (By the way, there were some other questions I asked about the PEP 382 revisions, that you didn't reply to in previous emails (such as the format of the strings to be returned by find_path()); I hope either you or Martin can fill those in for me, and hopefully update the PEP with the things we have talked about in this thread.) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 382 progress: import hooks
On Thu, Jul 22, 2010 at 19:19, P.J. Eby p...@telecommunity.com wrote: At 01:51 PM 7/22/2010 +0100, Martin v. Löwis wrote: At EuroPython, I sat down with Brett and we propose an approach how namespace packages get along with import hooks. I reshuffled the order in which things get done a little bit, and added a section that elaborates on the hooks. Basically, a finder will need to support a find_path method, return all .pth files, and a loader will need to support a load_module_with_path method, to initialize __path__. Please comment if you think that this needs further changes; I'm not certain I understand it precisely. There seem to be some ambiguities in the spec, e.g.: If fullname is not found, is not a package, or does not have any *.pth files, None must be returned. What does is not a package actually mean in that context? The module is a module but not a package. What happens if an empty list is returned - does that mean the importer is saying, this is a package, whether it has an __init__.py or not? As for the list of strings returned, is each string the entire contents of the .pth file? Is it to be \n-separated, or is any universal-newlines-compatible string accepted? Is there a particular order in which .pth file contents are to be returned? Regarding load_module_with_path(), how does its specification differ from simply creating a module in sys.modules, setting its __path__, and then invoking the standard load_module()? (i.e., is this method actually needed, since a correct PEP 302 loader *must* reuse an existing module object in sys.modules) It must reuse the module itself but a proper reload would reset __path__ as leaving it unchanged is not a proper resetting of the module object. So this module is needed in order to force the loader I'll hope to start implementing it soon. Am I correct in understanding that, as written, one would have to redefine __import__ to implement this in a library for older Python versions? Or is it implementable as a meta_path importer? Redefine __import__ (unless Martin and I are missing something, but I tried to think of how to implement this use sys.meta_path and couldn't come up with a solution). -Brett Regards, Martin Thanks for your work on this, I was just thinking about pinging to see how it was going. ;-) (I want setuptools 0.7 to be able to supply an add-on module for supporting this PEP in older Pythons, so that its current .pth hacks for implementing namespace packages can be dropped.) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/brett%40python.org ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 382 progress: import hooks
On 7/23/2010 11:57 AM, Brett Cannon wrote: On Thu, Jul 22, 2010 at 19:19, P.J. Eby p...@telecommunity.com mailto:p...@telecommunity.com wrote: At 01:51 PM 7/22/2010 +0100, Martin v. Löwis wrote: At EuroPython, I sat down with Brett and we propose an approach how namespace packages get along with import hooks. I reshuffled the order in which things get done a little bit, and added a section that elaborates on the hooks. Basically, a finder will need to support a find_path method, return all .pth files, and a loader will need to support a load_module_with_path method, to initialize __path__. Please comment if you think that this needs further changes; I'm not certain I understand it precisely. There seem to be some ambiguities in the spec, e.g.: If fullname is not found, is not a package, or does not have any *.pth files, None must be returned. What does is not a package actually mean in that context? The module is a module but not a package. so s/is not a package/is a module rather than a package/ perhaps? regards Steve -- Steve Holden +1 571 484 6266 +1 800 494 3119 DjangoCon US September 7-9, 2010http://djangocon.us/ See Python Video! http://python.mirocommunity.org/ Holden Web LLC http://www.holdenweb.com/ ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 382 progress: import hooks
At 11:57 AM 7/23/2010 +0100, Brett Cannon wrote: On Thu, Jul 22, 2010 at 19:19, P.J. Eby mailto:p...@telecommunity.comp...@telecommunity.com wrote: What does is not a package actually mean in that context? The module is a module but not a package. Um... that's not any clearer. Are you saying that a module of the same name takes precedence over a package? Is that the current precedence as well? Regarding load_module_with_path(), how does its specification differ from simply creating a module in sys.modules, setting its __path__, and then invoking the standard load_module()?  (i.e., is this method actually needed, since a correct PEP 302 loader *must* reuse an existing module object in sys.modules) It must reuse the module itself but a proper reload would reset __path__ as leaving it unchanged is not a proper resetting of the module object. So this module is needed in order to force the loader Um, no. Reloading doesn't reset the module contents, not even __path__. Never has, from Python 2.2 through 2.7 -- even in 3.1. At least, not for normal filesystem .py/.pyc files. (I tested with 'os', adding an extra 'foo' attribute, and also setting a __path__; both were unaffected by reload(), in all 7 Python versions. Perhaps you're saying this happens with zipfiles, or packages that already have a __path__, or...?  Am I correct in understanding that, as written, one would have to redefine __import__ to implement this in a library for older Python versions?  Or is it implementable as a meta_path importer? Redefine __import__ (unless Martin and I are missing something, but I tried to think of how to implement this use sys.meta_path and couldn't come up with a solution). I'm thinking it *could* be done with a meta_path hook, but only by doubling the search length in the event that the search failed. That seems a bit icky, but replacing the entire import process seems ickier (more code surface to maintain, more bug potential) in the case of supporting older Pythons. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 382 progress: import hooks
At 01:51 PM 7/22/2010 +0100, Martin v. Löwis wrote: At EuroPython, I sat down with Brett and we propose an approach how namespace packages get along with import hooks. I reshuffled the order in which things get done a little bit, and added a section that elaborates on the hooks. Basically, a finder will need to support a find_path method, return all .pth files, and a loader will need to support a load_module_with_path method, to initialize __path__. Please comment if you think that this needs further changes; I'm not certain I understand it precisely. There seem to be some ambiguities in the spec, e.g.: If fullname is not found, is not a package, or does not have any *.pth files, None must be returned. What does is not a package actually mean in that context? What happens if an empty list is returned - does that mean the importer is saying, this is a package, whether it has an __init__.py or not? As for the list of strings returned, is each string the entire contents of the .pth file? Is it to be \n-separated, or is any universal-newlines-compatible string accepted? Is there a particular order in which .pth file contents are to be returned? Regarding load_module_with_path(), how does its specification differ from simply creating a module in sys.modules, setting its __path__, and then invoking the standard load_module()? (i.e., is this method actually needed, since a correct PEP 302 loader *must* reuse an existing module object in sys.modules) I'll hope to start implementing it soon. Am I correct in understanding that, as written, one would have to redefine __import__ to implement this in a library for older Python versions? Or is it implementable as a meta_path importer? Regards, Martin Thanks for your work on this, I was just thinking about pinging to see how it was going. ;-) (I want setuptools 0.7 to be able to supply an add-on module for supporting this PEP in older Pythons, so that its current .pth hacks for implementing namespace packages can be dropped.) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
