Re: [Python-Dev] PEP 382: Namespace Packages
P.J. Eby wrote: I didn't say there's *no* desire, however IIRC the only person who *ever* asked on distutils-sig how to do a base package with setuptools was the author of the ll.* packages. I've asked before ;-) Chris -- Simplistix - Content Management, Zope Python Consulting - http://www.simplistix.co.uk ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 382: Namespace Packages
Martin v. Löwis wrote: I, for one, have been trying to figure out how to do base namespace packages for years... You mean, without PEP 382? That won't be possible, unless you can coordinate all addon packages. Base packages are a feature solely of PEP 382. Marc-Andre has achieved this, I think, without the PEP, but I never really understood how :-S Chris -- Simplistix - Content Management, Zope Python Consulting - http://www.simplistix.co.uk ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 382: Namespace Packages
-On [20090501 20:59], Martin v. Löwis (mar...@v.loewis.de) wrote: Right: if all portions install into the same directory, you can have base packages already. Speaking as a user of packages, this use case is one I hardly ever encounter with the Python software/modules/packages I use. The only ones that spring to mind are the mx.* and ll.* packages. The rest simply create their own namespace as package.*, but there's nothing that uses that same namespace and installs separately from the base package that I know of. -- Jeroen Ruigrok van der Werven asmodai(-at-)in-nomine.org / asmodai イェルーン ラウフロック ヴァン デル ウェルヴェン http://www.in-nomine.org/ | http://www.rangaku.org/ | GPG: 2EAC625B Knowledge was inherent in all things. The world was a library... ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 382: Namespace Packages
Right: if all portions install into the same directory, you can have base packages already. Speaking as a user of packages, this use case is one I hardly ever encounter with the Python software/modules/packages I use. The only ones that spring to mind are the mx.* and ll.* packages. The rest simply create their own namespace as package.*, but there's nothing that uses that same namespace and installs separately from the base package that I know of. There are a few others, though: zope.*, repoze.*, redturtle.*, iw.*, plone.*, pycopia.*, p4a.*, plonehrm.*, plonetheme.*, pbp.*, lovely.*, xm.*, paste.*, Products.*, buildout.*, five.*, silva.*, tl.*, tw.*, themerubber.*, themetweaker.*, zc.*, z3c.*, zgeo.*, z3ext.*, etc. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 382: Namespace Packages
-On [20090509 13:40], Martin v. Löwis (mar...@v.loewis.de) wrote: There are a few others, though: zope.*, repoze.*, redturtle.*, iw.*, plone.*, pycopia.*, p4a.*, plonehrm.*, plonetheme.*, pbp.*, lovely.*, xm.*, paste.*, Products.*, buildout.*, five.*, silva.*, tl.*, tw.*, themerubber.*, themetweaker.*, zc.*, z3c.*, zgeo.*, z3ext.*, etc. Can be fairly said, though, that the majority of those you just named are related to Zope? That would explain why I won't know of them as I avoid Zope like the plague. -- Jeroen Ruigrok van der Werven asmodai(-at-)in-nomine.org / asmodai イェルーン ラウフロック ヴァン デル ウェルヴェン http://www.in-nomine.org/ | http://www.rangaku.org/ | GPG: 2EAC625B Hope is a letter that never arrives, delivered by the postman of my fear... ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 382: Namespace Packages
-On [20090509 16:07], Chris Withers (ch...@simplistix.co.uk) wrote: They're also all pure namespace packages rather than base + addons, which is what we've been discussing... But from Martin's email I understood it more as being base packages. Unless I misunderstood, of course. If correct, which is it? More fool you... Maybe, used/worked with it and don't care for it one iota. But that's a whole different discussion. -- Jeroen Ruigrok van der Werven asmodai(-at-)in-nomine.org / asmodai イェルーン ラウフロック ヴァン デル ウェルヴェン http://www.in-nomine.org/ | http://www.rangaku.org/ | GPG: 2EAC625B Naritai jibun wo surikaetemo egao wa itsudemo suteki desuka... ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 382: Namespace Packages
Jeroen Ruigrok van der Werven wrote: -On [20090509 13:40], Martin v. Löwis (mar...@v.loewis.de) wrote: There are a few others, though: zope.*, repoze.*, redturtle.*, iw.*, plone.*, pycopia.*, p4a.*, plonehrm.*, plonetheme.*, pbp.*, lovely.*, xm.*, paste.*, Products.*, buildout.*, five.*, silva.*, tl.*, tw.*, themerubber.*, themetweaker.*, zc.*, z3c.*, zgeo.*, z3ext.*, etc. Can be fairly said, though, that the majority of those you just named are related to Zope? They're also all pure namespace packages rather than base + addons, which is what we've been discussing... That would explain why I won't know of them as I avoid Zope like the plague. More fool you... Chris -- Simplistix - Content Management, Zope Python Consulting - http://www.simplistix.co.uk ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 382: Namespace Packages
Jeroen Ruigrok van der Werven wrote: -On [20090509 16:07], Chris Withers (ch...@simplistix.co.uk) wrote: They're also all pure namespace packages rather than base + addons, which is what we've been discussing... But from Martin's email I understood it more as being base packages. Unless I misunderstood, of course. If correct, which is it? The list I gave you was a list of distributions that include namespace packages (using the setuptools mechanism). I don't think that any of them has the notion of a base package, as the setuptools mechanism doesn't support base packages. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 382: Namespace Packages
P.J. Eby wrote: At 06:15 PM 4/15/2009 +0200, M.-A. Lemburg wrote: The much more common use case is that of wanting to have a base package installation which optional add-ons that live in the same logical package namespace. Please see the large number of Zope and PEAK distributions on PyPI as minimal examples that disprove this being the common use case. If you mean the common use case as opposed to having code in the __init__.py of the namespace package, I think you'll find that's because people (especially me!) don't know how to do this, not because we don't want to! Chris - who would actually like to know how to do this, with or without the PEP, and how to indicate interdependencies in situations like this to setuptools... -- Simplistix - Content Management, Zope Python Consulting - http://www.simplistix.co.uk ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 382: Namespace Packages
P.J. Eby wrote: It's unclear, however, who is using base packages besides mx.* and ll.*, although I'd guess from the PyPI listings that perhaps Django is. (It seems that base packages are more likely to use a 'base-extension' naming pattern, vs. the 'namespace.project' pattern used by pure packages.) I'll stress it again in case you missed it the first time: I think the main reason people use pure namespace versus base namespace packages is because hardly anyone know how to do the latter, not because there is no desire to do so! I, for one, have been trying to figure out how to do base namespace packages for years... Chris -- Simplistix - Content Management, Zope Python Consulting - http://www.simplistix.co.uk ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 382: Namespace Packages
It's unclear, however, who is using base packages besides mx.* and ll.*, although I'd guess from the PyPI listings that perhaps Django is. (It seems that base packages are more likely to use a 'base-extension' naming pattern, vs. the 'namespace.project' pattern used by pure packages.) I'll stress it again in case you missed it the first time: I think the main reason people use pure namespace versus base namespace packages is because hardly anyone know how to do the latter, not because there is no desire to do so! I, for one, have been trying to figure out how to do base namespace packages for years... You mean, without PEP 382? That won't be possible, unless you can coordinate all addon packages. Base packages are a feature solely of PEP 382. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 382: Namespace Packages
At 05:35 PM 5/1/2009 +0100, Chris Withers wrote: P.J. Eby wrote: It's unclear, however, who is using base packages besides mx.* and ll.*, although I'd guess from the PyPI listings that perhaps Django is. (It seems that base packages are more likely to use a 'base-extension' naming pattern, vs. the 'namespace.project' pattern used by pure packages.) I'll stress it again in case you missed it the first time: I think the main reason people use pure namespace versus base namespace packages is because hardly anyone know how to do the latter, not because there is no desire to do so! I didn't say there's *no* desire, however IIRC the only person who *ever* asked on distutils-sig how to do a base package with setuptools was the author of the ll.* packages. And in the case of at least the zope.* peak.* and osaf.* namespace packages it was specifically *not* the intention to have a base __init__. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 382: Namespace Packages
At 07:41 PM 5/1/2009 +0200, Martin v. Löwis wrote: It's unclear, however, who is using base packages besides mx.* and ll.*, although I'd guess from the PyPI listings that perhaps Django is. (It seems that base packages are more likely to use a 'base-extension' naming pattern, vs. the 'namespace.project' pattern used by pure packages.) I'll stress it again in case you missed it the first time: I think the main reason people use pure namespace versus base namespace packages is because hardly anyone know how to do the latter, not because there is no desire to do so! I, for one, have been trying to figure out how to do base namespace packages for years... You mean, without PEP 382? That won't be possible, unless you can coordinate all addon packages. Base packages are a feature solely of PEP 382. Actually, if you are using only the distutils, you can do this by listing only modules in the addon projects; this is how the ll.* tools are doing it. That only works if the packages are all being installed in the same directory, though, not as eggs. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 382: Namespace Packages
Actually, if you are using only the distutils, you can do this by listing only modules in the addon projects; this is how the ll.* tools are doing it. That only works if the packages are all being installed in the same directory, though, not as eggs. Right: if all portions install into the same directory, you can have base packages already. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 382: Namespace Packages
On 04:56 am, p...@telecommunity.com wrote: At 03:58 AM 4/17/2009 +, gl...@divmod.com wrote: Just as a use-case: would the Java com.* namespace be an example of a pure package with no base? i.e. lots of projects are in it, but no project owns it? Er, I suppose. I was thinking more of the various 'com.foo' and 'org.bar' packages as being the pure namespaces in question. For Python, a flat is better than nested approach seems fine at the moment. Sure. I wasn't saying we should go down the domain-names-are-package- names road for Python itself, just that com.* is a very broad example of a multi-vendor namespace :). Entries on __path__ slow down import, so my understanding of the platonic ideal of a system python installation is one which has a single directory where all packages reside, and a set of metadata off to the side explaining which files belong to which distributions so they can be uninstalled by a package manager. True... except that part of the function of the PEP is to ensure that if you install those separately-distributed modules to the same directory, it still needs to work as a package and not have any inter- package file conflicts. Are you just referring to anything other than the problem of multiple packages overwriting __init__.py here? It's phrased in a very general way that makes me think maybe there's something else going on. So another clarification I'd like in the PEP is an explanation of motivation. For example, it comes as a complete surprise to me that the expectation of namespace packages was to provide only single- source namespaces like zope.*, peak.*, twisted.*. As I mentioned above, I implicitly thought this was more for com.*, twisted.plugins.*. Well, aside from twisted.plugins, I wasn't aware of anybody in Python doing that... and as I described, I never really interpreted that through the lens of namespace package vs. plugin finding. There is some overlap. In particular, in the vendor distribution case, I would like there to be one nice, declarative Python way to say please put these modules into the same package. In the past, Debian in particular has produced some badly broken Twisted packages in the past because there was no standard Python way to say I have some modules here that go into an existing package. Since every distribution has its own funny ideas about what the filesystem should look like, this has come up for us in a variety of ways. I'd like it if we could use the official way of declaring a namespace package for that. Right now it just says that it's a package which resides in multiple directories, and it's not made clear why that's a desirable feature. Good point; perhaps you can suggest some wording on these matters to Martin? I think the thing I said in my previous message about multiple distributions is a good start. That might not be everything, but I think it's clearly the biggest motivation. Okay. So what I'm hearing is that Twisted should happily continue using our own wacky __path__-calculation logic for twisted.plugins, but that *twisted* should be a namespace package so that our separate distributions (TwistedCore, TwistedWeb, TwistedConch, et. al.) can be installed into separate directories. Yes. I'm fairly happy with that, except the aforementioned communication- channel-with-packagers feature of namespace packages; they unambiguously say multiple OS packages may contribute modules to this Python package. Thanks for taking the time to participate in this and add another viewpoint to the mix, not to mention clarifying some areas where the PEP could be clearer. My pleasure. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 382: Namespace Packages
At 03:46 AM 4/16/2009 +, gl...@divmod.com wrote: On 15 Apr, 09:11 pm, p...@telecommunity.com wrote: I think that there is some confusion here. A main package or buildout that assembles a larger project from components is not the same thing as having a base package for a namespace package. I'm certainly confused. Twisted has its own system for namespace packages, and I'm not really sure where we fall in this discussion. I haven't been able to follow the whole thread, but my original understanding was that the PEP supports defining packages, which we now seem to be calling base packages, just fine. Yes, it does. The discussion since the original proposal, however, has been dominated by MAL's counterproposal, which *requires* a defining package. There is a slight distinction between base package and defining package, although I suppose I've been using them a bit interchangeably. Base package describes a use case: you have a base package which is extended in the same namespace. In that use case, you may want to place your base package in the defining package. In contrast, setuptools does not support a defining package, so if you have a base package, you must place it in a submodule or subpackage of the namespace. Does that all make sense now? MAL's proposal requires a defining package, which is counterproductive if you have a pure package with no base, since it now requires you to create an additional project on PyPI just to hold your defining package. I'd appreciate it if the PEP could also be extended cover Twisted's very similar mechanism for namespace packages, twisted.plugin.pluginPackagePaths. I know this is not quite as widely used as setuptools' namespace package support, but its existence belies a need for standardization. The PEP also seems a bit vague with regard to the treatment of other directories containing __init__.py and *.pkg files. Do you have a clarification to suggest? My understanding (probably a projection) is that to be a nested namespace package, you have to have a parent namespace package. The concept of a defining package seems important to avoid conflicts like this one: http://twistedmatrix.com/trac/ticket/2339 More specifically I don't quite understand the PEP's intentions towards hierarchical packages. It says that all of sys.path will be searched, but what about this case? In Twisted, the suggested idiom to structure a project which wants to provide Twisted plugins is to have a directory structure like this: MyProject/ myproject/ __init__.py twisted/ plugins/ myproject_plugin.py If you then put MyProject on PYTHONPATH, MyProject/twisted/plugins will be picked up automatically by the plugin machinery. Namespaces are not plugins and vice versa. The purpose of a namespace package is to allow projects managed by the same entity to share a namespace (ala Java package names) and avoid naming conflicts with other authors. A plugin system, by contrast, is explicitly intended for use by multiple authors, so the use case is rather different... and using namespace packages for plugins actually *increases* the possibility of naming conflicts, unless you add back in another level of hierarchy. (As apparently you are recommending via myproject_plugin.) However, as twisted is *not* a namespace package in the same way, .py files in MyProject/twisted/ would not be picked up - this is very much intentional, since the twisted namespace is intended to be reserved for packages that we actually produce. If either MyProject/twisted or MyProject/twisted/plugins/ had an __init__.py, then no modules in MyProject/twisted/plugins/ would be picked up, because it would be considered a conflicting package. Precisely. Note, however, that neither is twisted.plugins a namespace package, and it should not contain any .pkg files. I don't think it's reasonable to abuse PEP 382 namespace packages as a plugin system. In setuptools' case, a different mechanism is provided for locating plugin code, and of course Twisted already has its own system for the same thing. It would be nice to have a standardized way of locating plugins in the stdlib, but that will need to be a different PEP. I hope this all makes sense. As I understand it, both setuptools and the proposed standard would either still have the bug described by ticket 2339 above, or would ignore twisted/plugins/ as a namespace package because its parent isn't a namespace package. If twisted/ lacked an __init__.py, then setuptools would ignore it. Under PEP 382, the same, unless it had .pkg files. (Again, setuptools explicitly does not support using namespace packages as a plugin mechanism.) P.S.: vendor packaging systems *ARE* a major use case for just about any aspect of Python's package structure. I really liked MvL's coverage of vendor packages, in the PEP, since this could equally well apply to
Re: [Python-Dev] PEP 382: Namespace Packages
On 16 Apr, 03:36 pm, p...@telecommunity.com wrote: At 03:46 AM 4/16/2009 +, gl...@divmod.com wrote: On 15 Apr, 09:11 pm, p...@telecommunity.com wrote: Twisted has its own system for namespace packages, and I'm not really sure where we fall in this discussion. I haven't been able to follow the whole thread, but my original understanding was that the PEP supports defining packages, which we now seem to be calling base packages, just fine. Yes, it does. The discussion since the original proposal, however, has been dominated by MAL's counterproposal, which *requires* a defining package. [snip clarifications] Does that all make sense now? Yes. Thank you very much for the detailed explanation. It was more than I was due :-). MAL's proposal requires a defining package, which is counterproductive if you have a pure package with no base, since it now requires you to create an additional project on PyPI just to hold your defining package. Just as a use-case: would the Java com.* namespace be an example of a pure package with no base? i.e. lots of projects are in it, but no project owns it? I'd appreciate it if the PEP could also be extended cover Twisted's very similar mechanism for namespace packages, twisted.plugin.pluginPackagePaths. I know this is not quite as widely used as setuptools' namespace package support, but its existence belies a need for standardization. The PEP also seems a bit vague with regard to the treatment of other directories containing __init__.py and *.pkg files. Do you have a clarification to suggest? My understanding (probably a projection) is that to be a nested namespace package, you have to have a parent namespace package. Just to clarify things on my end: namespace package to *me* means package with modules provided from multiple distributions (the distutils term). The definition provided by the PEP, that a package is spread over multiple directories on disk, seems like an implementation detail. Entries on __path__ slow down import, so my understanding of the platonic ideal of a system python installation is one which has a single directory where all packages reside, and a set of metadata off to the side explaining which files belong to which distributions so they can be uninstalled by a package manager. Of course, for a development installation, easy uninstallation and quick swapping between different versions of relevant dependencies is more important than good import performance. So in that case, you would want to optimize differently by having all of your distributions installed into separate directories, with a long PYTHONPATH or lots of .pth files to point at them. And of course you may want a hybrid of the two. So another clarification I'd like in the PEP is an explanation of motivation. For example, it comes as a complete surprise to me that the expectation of namespace packages was to provide only single-source namespaces like zope.*, peak.*, twisted.*. As I mentioned above, I implicitly thought this was more for com.*, twisted.plugins.*. Right now it just says that it's a package which resides in multiple directories, and it's not made clear why that's a desirable feature. The concept of a defining package seems important to avoid conflicts like this one: http://twistedmatrix.com/trac/ticket/2339 [snip some stuff about plugins and package layout] Namespaces are not plugins and vice versa. The purpose of a namespace package is to allow projects managed by the same entity to share a namespace (ala Java package names) and avoid naming conflicts with other authors. I think this is missing a key word: *separate* projects managed by the same entity. Hmm. I thought I could illustrate that the same problem actually occurs without using a plugin system, but I can actually only come up with an example if an application implements multi-library-version compatibility by doing try: from bad_old_name import bad_old_feature as feature except ImportError: from good_new_name import good_new_feature as feature rather than the other way around; and that's a terrible idea for other reasons. Other than that, you'd have to use pkg_resources.resource_listdir or somesuch, at which point you pretty much are implementing a plugin system. So I started this reply disagreeing but I think I've convinced myself that you're right. Precisely. Note, however, that neither is twisted.plugins a namespace package, and it should not contain any .pkg files. I don't think it's reasonable to abuse PEP 382 namespace packages as a plugin system. In setuptools' case, a different mechanism is provided for locating plugin code, and of course Twisted already has its own system for the same thing. It would be nice to have a standardized way of locating plugins in the stdlib, but that will need to be a different PEP. Okay. So what I'm hearing is that Twisted should happily
Re: [Python-Dev] PEP 382: Namespace Packages
At 03:58 AM 4/17/2009 +, gl...@divmod.com wrote: Just as a use-case: would the Java com.* namespace be an example of a pure package with no base? i.e. lots of projects are in it, but no project owns it? Er, I suppose. I was thinking more of the various 'com.foo' and 'org.bar' packages as being the pure namespaces in question. For Python, a flat is better than nested approach seems fine at the moment. Just to clarify things on my end: namespace package to *me* means package with modules provided from multiple distributions (the distutils term). The definition provided by the PEP, that a package is spread over multiple directories on disk, seems like an implementation detail. Agreed. Entries on __path__ slow down import, so my understanding of the platonic ideal of a system python installation is one which has a single directory where all packages reside, and a set of metadata off to the side explaining which files belong to which distributions so they can be uninstalled by a package manager. True... except that part of the function of the PEP is to ensure that if you install those separately-distributed modules to the same directory, it still needs to work as a package and not have any inter-package file conflicts. Of course, for a development installation, easy uninstallation and quick swapping between different versions of relevant dependencies is more important than good import performance. So in that case, you would want to optimize differently by having all of your distributions installed into separate directories, with a long PYTHONPATH or lots of .pth files to point at them. And of course you may want a hybrid of the two. Yep. So another clarification I'd like in the PEP is an explanation of motivation. For example, it comes as a complete surprise to me that the expectation of namespace packages was to provide only single-source namespaces like zope.*, peak.*, twisted.*. As I mentioned above, I implicitly thought this was more for com.*, twisted.plugins.*. Well, aside from twisted.plugins, I wasn't aware of anybody in Python doing that... and as I described, I never really interpreted that through the lens of namespace package vs. plugin finding. Right now it just says that it's a package which resides in multiple directories, and it's not made clear why that's a desirable feature. Good point; perhaps you can suggest some wording on these matters to Martin? Okay. So what I'm hearing is that Twisted should happily continue using our own wacky __path__-calculation logic for twisted.plugins, but that *twisted* should be a namespace package so that our separate distributions (TwistedCore, TwistedWeb, TwistedConch, et. al.) can be installed into separate directories. Yes. Thanks for taking the time to participate in this and add another viewpoint to the mix, not to mention clarifying some areas where the PEP could be clearer. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 382: Namespace Packages
On 2009-04-15 02:32, P.J. Eby wrote: At 10:59 PM 4/14/2009 +0200, M.-A. Lemburg wrote: You are missing the point: When breaking up a large package that lives in site-packages into smaller distribution bundles, you don't need namespace packages at all, so the PEP doesn't apply. The way this works is by having a base distribution bundle that includes the needed __init__.py file and a set of extension bundles the add other files to the same directory (without including another copy of __init__.py). The extension bundles include a dependency on the base package to make sure that it always gets installed first. If we're going to keep that practice, there's no point to having the PEP: all three methods (base+extensions, pkgutil, setuptools) all work just fine as they are, with no changes to importing or the stdlib. Again: the PEP is about creating a standard for namespace packages. It's not about making namespace packages easy to use for Linux distribution maintainers. Instead, it's targeting *developers* that want to enable shipping a single package in multiple, separate pieces, giving the user the freedom to the select the ones she needs. Of course, this is possible today using various other techniques. The point is that there is no standard for namespace packages and that's what the PEP is trying to solve. In particular, without the feature of being able to drop that practice, there would be no reason for setuptools to adopt the PEP. That's why I'm -1 on your proposal: it's actually inferior to the methods we already have today. It's simpler and more in line with the Python Zen, not inferior. You are free not to support it in setuptools - the methods implemented in setuptools will continue to work as they are, but continue to require support code and, over time, no longer be compatible with other tools building upon the standard defined in the PEP. In the end, it's the user that decides: whether to go with a standard or not. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Apr 15 2009) Python/Zope Consulting and Support ...http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ 2009-03-19: Released mxODBC.Connect 1.0.1 http://python.egenix.com/ ::: Try our new mxODBC.Connect Python Database Interface for free ! eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 382: Namespace Packages
At 09:51 AM 4/15/2009 +0200, M.-A. Lemburg wrote: On 2009-04-15 02:32, P.J. Eby wrote: At 10:59 PM 4/14/2009 +0200, M.-A. Lemburg wrote: You are missing the point: When breaking up a large package that lives in site-packages into smaller distribution bundles, you don't need namespace packages at all, so the PEP doesn't apply. The way this works is by having a base distribution bundle that includes the needed __init__.py file and a set of extension bundles the add other files to the same directory (without including another copy of __init__.py). The extension bundles include a dependency on the base package to make sure that it always gets installed first. If we're going to keep that practice, there's no point to having the PEP: all three methods (base+extensions, pkgutil, setuptools) all work just fine as they are, with no changes to importing or the stdlib. Again: the PEP is about creating a standard for namespace packages. It's not about making namespace packages easy to use for Linux distribution maintainers. Instead, it's targeting *developers* that want to enable shipping a single package in multiple, separate pieces, giving the user the freedom to the select the ones she needs. Of course, this is possible today using various other techniques. The point is that there is no standard for namespace packages and that's what the PEP is trying to solve. In particular, without the feature of being able to drop that practice, there would be no reason for setuptools to adopt the PEP. That's why I'm -1 on your proposal: it's actually inferior to the methods we already have today. It's simpler and more in line with the Python Zen, not inferior. You are free not to support it in setuptools - the methods implemented in setuptools will continue to work as they are, but continue to require support code and, over time, no longer be compatible with other tools building upon the standard defined in the PEP. In the end, it's the user that decides: whether to go with a standard or not. Up until this point, I've been trying to help you understand the use cases, but it's clear now that you already understand them, you just don't care. That wouldn't be a problem if you just stayed on the sidelines, instead of actively working to make those use cases more difficult for everyone else than they already are. Anyway, since you clearly understand precisely what you're doing, I'm now going to stop trying to explain things, as my responses are apparently just encouraging you, and possibly convincing bystanders that there's some genuine controversy here as well. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 382: Namespace Packages
[much quote-trimming, the following is intended to just give the gist, but the bits quoted below are not in directe response to each other] On Wed, Apr 15, 2009, P.J. Eby wrote: At 09:51 AM 4/15/2009 +0200, M.-A. Lemburg wrote: [...] Again: the PEP is about creating a standard for namespace packages. It's not about making namespace packages easy to use for Linux distribution maintainers. Instead, it's targeting *developers* that want to enable shipping a single package in multiple, separate pieces, giving the user the freedom to the select the ones she needs. [...] [...] Anyway, since you clearly understand precisely what you're doing, I'm now going to stop trying to explain things, as my responses are apparently just encouraging you, and possibly convincing bystanders that there's some genuine controversy here as well. For the benefit of us bystanders, could you summarize your vote at this point? Given the PEP's intended goals, if you do not oppose the PEP, are there any changes you think should be made? -- Aahz (a...@pythoncraft.com) * http://www.pythoncraft.com/ Why is this newsgroup different from all other newsgroups? ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 382: Namespace Packages
On 2009-04-15 16:44, P.J. Eby wrote: At 09:51 AM 4/15/2009 +0200, M.-A. Lemburg wrote: On 2009-04-15 02:32, P.J. Eby wrote: At 10:59 PM 4/14/2009 +0200, M.-A. Lemburg wrote: You are missing the point: When breaking up a large package that lives in site-packages into smaller distribution bundles, you don't need namespace packages at all, so the PEP doesn't apply. The way this works is by having a base distribution bundle that includes the needed __init__.py file and a set of extension bundles the add other files to the same directory (without including another copy of __init__.py). The extension bundles include a dependency on the base package to make sure that it always gets installed first. If we're going to keep that practice, there's no point to having the PEP: all three methods (base+extensions, pkgutil, setuptools) all work just fine as they are, with no changes to importing or the stdlib. Again: the PEP is about creating a standard for namespace packages. It's not about making namespace packages easy to use for Linux distribution maintainers. Instead, it's targeting *developers* that want to enable shipping a single package in multiple, separate pieces, giving the user the freedom to the select the ones she needs. Of course, this is possible today using various other techniques. The point is that there is no standard for namespace packages and that's what the PEP is trying to solve. In particular, without the feature of being able to drop that practice, there would be no reason for setuptools to adopt the PEP. That's why I'm -1 on your proposal: it's actually inferior to the methods we already have today. It's simpler and more in line with the Python Zen, not inferior. You are free not to support it in setuptools - the methods implemented in setuptools will continue to work as they are, but continue to require support code and, over time, no longer be compatible with other tools building upon the standard defined in the PEP. In the end, it's the user that decides: whether to go with a standard or not. Up until this point, I've been trying to help you understand the use cases, but it's clear now that you already understand them, you just don't care. That wouldn't be a problem if you just stayed on the sidelines, instead of actively working to make those use cases more difficult for everyone else than they already are. Anyway, since you clearly understand precisely what you're doing, I'm now going to stop trying to explain things, as my responses are apparently just encouraging you, and possibly convincing bystanders that there's some genuine controversy here as well. Hopefully, bystanders will understand that the one single use case you are always emphasizing, namely that of Linux distribution maintainers trying to change the package installation layout, is really a rather uncommon and rare use case. It is true that I do understand what the namespace package idea is all about. I've been active in Python package development since they were first added to Python as a new built-in import feature in Python 1.5 and have been distributing packages with package add-ons for more than a decade... For some history, have a look at: http://www.python.org/doc/essays/packages.html Also note how that essay discourages the use of .pth files: If the package really requires adding one or more directories on sys.path (e.g. because it has not yet been structured to support dotted-name import), a path configuration file named package.pth can be placed in either the site-python or site-packages directory. ... A typical installation should have no or very few .pth files or something is wrong, and if you need to play with the search order, something is very wrong. Back to the PEP: The much more common use case is that of wanting to have a base package installation which optional add-ons that live in the same logical package namespace. The PEP provides a way to solve this use case by giving both developers and users a standard at hand which they can follow without having to rely on some non-standard helpers and across Python implementations. My proposal tries to solve this without adding yet another .pth file like mechanism - hopefully in the spirit of the original Python package idea. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Apr 15 2009) Python/Zope Consulting and Support ...http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ 2009-03-19: Released mxODBC.Connect 1.0.1 http://python.egenix.com/ ::: Try our new mxODBC.Connect Python Database Interface for free ! eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO
Re: [Python-Dev] PEP 382: Namespace Packages
At 09:10 AM 4/15/2009 -0700, Aahz wrote: For the benefit of us bystanders, could you summarize your vote at this point? Given the PEP's intended goals, if you do not oppose the PEP, are there any changes you think should be made? I'm +1 on Martin's original version of the PEP, subject to the point brought up by someone that .pkg should be changed to a different extension. I'm -1 on all of MAL's proposed revisions, as IMO they are a step backwards: they standardize an approach that will create problems that don't need to exist, and don't exist now. Martin's proposal is an improvement on the status quo, Marc's proposal is a dis-improvement. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 382: Namespace Packages
At 06:15 PM 4/15/2009 +0200, M.-A. Lemburg wrote: The much more common use case is that of wanting to have a base package installation which optional add-ons that live in the same logical package namespace. Please see the large number of Zope and PEAK distributions on PyPI as minimal examples that disprove this being the common use case. I expect you will find a fair number of others, as well. In these cases, there is NO base package... the entire point of using namespace packages for these distributions is that a base package is neither necessary nor desirable. In other words, the base package scenario is the exception these days, not the rule. I actually know specifically of only one other such package besides your mx.* case, the logilab ll.* package. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 382: Namespace Packages
On Apr 15, 2009, at 12:15 PM, M.-A. Lemburg wrote: The much more common use case is that of wanting to have a base package installation which optional add-ons that live in the same logical package namespace. The PEP provides a way to solve this use case by giving both developers and users a standard at hand which they can follow without having to rely on some non-standard helpers and across Python implementations. I'm not sure I understand what advantage your proposal gives over the current mechanism for doing this. That is, add to your __init__.py file: from pkgutil import extend_path __path__ = extend_path(__path__, __name__) Can you describe the intended advantages over the status-quo a bit more clearly? James ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 382: Namespace Packages
On 2009-04-15 19:59, P.J. Eby wrote: At 06:15 PM 4/15/2009 +0200, M.-A. Lemburg wrote: The much more common use case is that of wanting to have a base package installation which optional add-ons that live in the same logical package namespace. Please see the large number of Zope and PEAK distributions on PyPI as minimal examples that disprove this being the common use case. I expect you will find a fair number of others, as well. In these cases, there is NO base package... the entire point of using namespace packages for these distributions is that a base package is neither necessary nor desirable. In other words, the base package scenario is the exception these days, not the rule. I actually know specifically of only one other such package besides your mx.* case, the logilab ll.* package. So now you're arguing against having base packages... at least you've dropped the strange idea of using Linux distribution maintainers as central use case ;-) Think of base namespace packages (the ones providing the __init__.py file) as defining the namespace. They setup ownership and the basic infrastructure needed by add-ons. If you take Zope as example, the Products/ package dir is a good example: the __init__.py file in that directory is provided by the Zope installation (generated during Zope instance creation), so Zope owns the package. With the proposal, Zope could declare this package dir a namespace base package by adding a __pkg__.py file to it. Zope add-ons could then be installed somewhere else on sys.path and include a Products/ dir as well, only this time it doesn't have the __init__.py file, but only a __pkg__.py file. Python would then take care of integrating the add-on Products/ dir Python module/package contents with the base package. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Apr 15 2009) Python/Zope Consulting and Support ...http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ 2009-03-19: Released mxODBC.Connect 1.0.1 http://python.egenix.com/ ::: Try our new mxODBC.Connect Python Database Interface for free ! eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 382: Namespace Packages
On Wed, Apr 15, 2009 at 01:59:34PM -0400, P.J. Eby wrote: Please see the large number of Zope and PEAK distributions on PyPI as minimal examples that disprove this being the common use case. I expect you will find a fair number of others, as well. ... In other words, the base package scenario is the exception these days, not the rule. I actually know specifically of only one other such package besides your mx.* case, the logilab ll.* package. Isn't that pretty even, then? zope.* and PEAK are two examples of one approach; and mx.* and ll.* are two examples that use the base package approach. Neither approach seems to be the more common one, and both are pretty rare. --amk ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 382: Namespace Packages
At 02:52 PM 4/15/2009 -0400, A.M. Kuchling wrote: On Wed, Apr 15, 2009 at 01:59:34PM -0400, P.J. Eby wrote: Please see the large number of Zope and PEAK distributions on PyPI as minimal examples that disprove this being the common use case. I expect you will find a fair number of others, as well. ... In other words, the base package scenario is the exception these days, not the rule. I actually know specifically of only one other such package besides your mx.* case, the logilab ll.* package. Isn't that pretty even, then? zope.* and PEAK are two examples of one approach; and mx.* and ll.* are two examples that use the base package approach. Neither approach seems to be the more common one, and both are pretty rare. If you view the package listings on PyPI, you'll see that the pure namespaces currently in use include: alchemist.* amplecode.* atomisator.* bda.* benri.* beyondskins.* bliptv.* bopen.* borg.* bud.* ... This is just going down to the 'b's, looking only at packages whose PyPI project name reflects a nested package name, and only including those with entries that: 1. use setuptools, 2. declare one or more namespace packages, and 3. do not depend on some sort of base or core package. Technically, setuptools doesn't support base packages anyway, but if the organization appeared to be based on a core+plugins/addons model (as opposed to collection of packages grouped in a namespace) I didn't include it in the list above -- i.e., I'm bending over backwards to be fair in the count. If somebody wants to do a formal count of base vs. pure, it might provide interesting stats. I initially only mentioned Zope and PEAK because I have direct knowledge of the developers' intent regarding their namespace packages. However, now that I've actually looked at a tiny sample of PyPI, it's clear that the actual field use of pure namespace packages has positively exploded since setuptools made it practical to use them. It's unclear, however, who is using base packages besides mx.* and ll.*, although I'd guess from the PyPI listings that perhaps Django is. (It seems that base packages are more likely to use a 'base-extension' naming pattern, vs. the 'namespace.project' pattern used by pure packages.) Of course, I am certainly not opposed to supporting base packages, and Martin's version of PEP 382 is a plus for setuptools because it would allow setuptools to better support the base scenario. But pure packages are definitely not a minority; in fact, a superficial observation of the full PyPI list suggests that there may be almost as many projects using pure-namespace packages, as there are non-namespaced projects! ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 382: Namespace Packages
On Wed, Apr 15, 2009 at 9:22 PM, P.J. Eby p...@telecommunity.com wrote: At 02:52 PM 4/15/2009 -0400, A.M. Kuchling wrote: On Wed, Apr 15, 2009 at 01:59:34PM -0400, P.J. Eby wrote: Please see the large number of Zope and PEAK distributions on PyPI as minimal examples that disprove this being the common use case. I expect you will find a fair number of others, as well. ... In other words, the base package scenario is the exception these days, not the rule. I actually know specifically of only one other such package besides your mx.* case, the logilab ll.* package. Isn't that pretty even, then? zope.* and PEAK are two examples of one approach; and mx.* and ll.* are two examples that use the base package approach. Neither approach seems to be the more common one, and both are pretty rare. If you view the package listings on PyPI, you'll see that the pure namespaces currently in use include: alchemist.* amplecode.* atomisator.* bda.* benri.* beyondskins.* bliptv.* bopen.* borg.* bud.* ... This is just going down to the 'b's, looking only at packages whose PyPI project name reflects a nested package name, and only including those with entries that: 1. use setuptools, 2. declare one or more namespace packages, and 3. do not depend on some sort of base or core package. Technically, setuptools doesn't support base packages anyway, but if the organization appeared to be based on a core+plugins/addons model (as opposed to collection of packages grouped in a namespace) I didn't include it in the list above -- i.e., I'm bending over backwards to be fair in the count. If somebody wants to do a formal count of base vs. pure, it might provide interesting stats. I initially only mentioned Zope and PEAK because I have direct knowledge of the developers' intent regarding their namespace packages. However, now that I've actually looked at a tiny sample of PyPI, it's clear that the actual field use of pure namespace packages has positively exploded since setuptools made it practical to use them. It's unclear, however, who is using base packages besides mx.* and ll.*, although I'd guess from the PyPI listings that perhaps Django is. (It seems that base packages are more likely to use a 'base-extension' naming pattern, vs. the 'namespace.project' pattern used by pure packages.) Of course, I am certainly not opposed to supporting base packages, and Martin's version of PEP 382 is a plus for setuptools because it would allow setuptools to better support the base scenario. But pure packages are definitely not a minority; in fact, a superficial observation of the full PyPI list suggests that there may be almost as many projects using pure-namespace packages, as there are non-namespaced projects! In the survey I have done on packaging, 34% of the people that answered are using setuptools namespace feature, which currently makes it impossible to use the namespace for the base package. Now for the base or core package, what peoplethat uses setuptools do most of the time: 1- they use zc.buildout so they don't need a base package : they list in a configuration files all packages needed to build the application, and one of these package happen to have the scripts to launch the application. 2 - they have a main package that doesn't use the same namespace, but uses setuptools instal_requires metadata to include namespaced packages. It acts like zc.buildout in some ways. For example, you mentioned atomisator.* in your example, this app has a main package called Atomisator (notice the upper A) that uses strategy #2 But frankly, the base package scenario is not spread these days simply because it's not obvious to do it without depending on a OS that has its own strategy to install packages. For example, if you are not under debian, it's a pain to use logilab packages because they use this common namespace for several packages and a plain python installation of the various packages won't work out of the box under other systems like windows. (and for pylint, I ended up creating my own distribution for windows...) So : - having namespaces natively in Python is a big win (Namespaces are one honking great idea -- let's do more of those!) - being able to still write some code under the primary namespace is something I (and lots of people) wish we could do with setuptools, so it's a big win too. Regards, Tarek -- Tarek Ziadé | http://ziade.org ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 382: Namespace Packages
On 2009-04-15 21:22, P.J. Eby wrote: At 02:52 PM 4/15/2009 -0400, A.M. Kuchling wrote: On Wed, Apr 15, 2009 at 01:59:34PM -0400, P.J. Eby wrote: Please see the large number of Zope and PEAK distributions on PyPI as minimal examples that disprove this being the common use case. I expect you will find a fair number of others, as well. ... In other words, the base package scenario is the exception these days, not the rule. I actually know specifically of only one other such package besides your mx.* case, the logilab ll.* package. Isn't that pretty even, then? zope.* and PEAK are two examples of one approach; and mx.* and ll.* are two examples that use the base package approach. Neither approach seems to be the more common one, and both are pretty rare. If you view the package listings on PyPI, you'll see that the pure namespaces currently in use include: alchemist.* amplecode.* atomisator.* bda.* benri.* beyondskins.* bliptv.* bopen.* borg.* bud.* ... This is just going down to the 'b's, looking only at packages whose PyPI project name reflects a nested package name, and only including those with entries that: 1. use setuptools, 2. declare one or more namespace packages, and 3. do not depend on some sort of base or core package. Technically, setuptools doesn't support base packages anyway, but if the organization appeared to be based on a core+plugins/addons model (as opposed to collection of packages grouped in a namespace) I didn't include it in the list above -- i.e., I'm bending over backwards to be fair in the count. Hmm, setuptools doesn't support the notion of base packages, ie. packages that provide their own __init__.py module, so I fail to see how your list or any other list of setuptools-depend packages can be taken as indicator for anything related to base packages. Since setuptools probably introduced the idea of namespace sharing packages to many authors in the first place, such a list is even less appropriate to use as sample base. That said, I don't think such statistics provide any useful information to decide on the namespace import strategy standard for Python which is the subject of the PEP. They just show that one helper-based mechanism is used more than others and that's simply a consequence of there not being a standard built-in way of using namespace packages in Python. Whether base packages are useful or not is really a side aspect of the PEP and my proposal. I'm more after a method that doesn't add more .pkg file cruft to Python's import mechanism. Those .pth files were originally meant to help older Python packages (think the early PIL or Numeric extensions) to integrate nicely into the new scheme without having to fully support dotted package names right from the start. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Apr 15 2009) Python/Zope Consulting and Support ...http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ 2009-03-19: Released mxODBC.Connect 1.0.1 http://python.egenix.com/ ::: Try our new mxODBC.Connect Python Database Interface for free ! eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 382: Namespace Packages
At 10:20 PM 4/15/2009 +0200, M.-A. Lemburg wrote: Whether base packages are useful or not is really a side aspect of the PEP and my proposal. It's not whether they're useful, it's whether they're required. Your proposal *requires* base packages, and for people who intend to use pure packages, this is NOT a feature: it's a bug. Specifically, it introduces a large number of unnecessary, boilerplate dependencies to their package distribution strategy. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 382: Namespace Packages
M.-A. Lemburg writes: Hmm, setuptools doesn't support the notion of base packages, ie. packages that provide their own __init__.py module, so I fail to see how your list or any other list of setuptools-depend packages can be taken as indicator for anything related to base packages. AFAICS the only things PJE has said about base packages is that (a) they aren't a universal use case for namespace packages, and (b) he'd like to be able to support them in setuptools, but admits that at present they aren't. Your arguments against the PEP supporting namespace packages as currently supported by setuptools seem purely theoretical to me, while he's defending an actual and common use case. Although practicality beats purity. I think that for this PEP it's more important to unify the various use cases for namespace packages than it is to get rid of the .pth files. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 382: Namespace Packages
At 09:59 AM 4/16/2009 +0900, Stephen J. Turnbull wrote: I think that for this PEP it's more important to unify the various use cases for namespace packages than it is to get rid of the .pth files. Actually, Martin's proposal *does* get rid of the .pth files in site-packages, and replaces them with other files inside the individual packages. (Thereby speeding startup times when many namespace packages are present but only a few are used.) So Martin's proposal is a win for performance and even for decreasing clutter. (The same number of special files will be present, but they will be moved inside the namespace package directories instead of being in the parent directory.) AFAICS the only things PJE has said about base packages is that (a) they aren't a universal use case for namespace packages, and (b) he'd like to be able to support them in setuptools, but admits that at present they aren't. ...and that Martin's proposal would actually permit me to do so, whereas MAL's proposal would not. Replacing __init__.py with a __pkg__.py wouldn't change any of the tradeoffs for how setuptools handles namespace packages, except to add an extra variable to consider (i.e., two filenames to keep track of). ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 382: Namespace Packages
On 15 Apr, 09:11 pm, p...@telecommunity.com wrote: I think that there is some confusion here. A main package or buildout that assembles a larger project from components is not the same thing as having a base package for a namespace package. I'm certainly confused. Twisted has its own system for namespace packages, and I'm not really sure where we fall in this discussion. I haven't been able to follow the whole thread, but my original understanding was that the PEP supports defining packages, which we now seem to be calling base packages, just fine. I don't understand the controversy over the counterproposal, since it seems roughly functionally equivalent to me. I'd appreciate it if the PEP could also be extended cover Twisted's very similar mechanism for namespace packages, twisted.plugin.pluginPackagePaths. I know this is not quite as widely used as setuptools' namespace package support, but its existence belies a need for standardization. The PEP also seems a bit vague with regard to the treatment of other directories containing __init__.py and *.pkg files. The concept of a defining package seems important to avoid conflicts like this one: http://twistedmatrix.com/trac/ticket/2339 More specifically I don't quite understand the PEP's intentions towards hierarchical packages. It says that all of sys.path will be searched, but what about this case? In Twisted, the suggested idiom to structure a project which wants to provide Twisted plugins is to have a directory structure like this: MyProject/ myproject/ __init__.py twisted/ plugins/ myproject_plugin.py If you then put MyProject on PYTHONPATH, MyProject/twisted/plugins will be picked up automatically by the plugin machinery. However, as twisted is *not* a namespace package in the same way, .py files in MyProject/twisted/ would not be picked up - this is very much intentional, since the twisted namespace is intended to be reserved for packages that we actually produce. If either MyProject/twisted or MyProject/twisted/plugins/ had an __init__.py, then no modules in MyProject/twisted/plugins/ would be picked up, because it would be considered a conflicting package. This is important so that users can choose not to load the system- installed Twisted's plugins when they have both a system-installed version of Twisted and a non-installed development version of Twisted found first on their PYTHONPATH, and switch between them to indicate which version they want to be the base or defining package for the twisted/plugins/ namespace. Developers might also want to have a system-installed Twisted, but a non-installed development version of MyProject on PYTHONPATH. I hope this all makes sense. As I understand it, both setuptools and the proposed standard would either still have the bug described by ticket 2339 above, or would ignore twisted/plugins/ as a namespace package because its parent isn't a namespace package. I apologize for not testing with current setuptools before asking, but I'm not sure my experiments would be valid given that my environment is set up with assumptions from Twisted's system. P.S.: vendor packaging systems *ARE* a major use case for just about any aspect of Python's package structure. I really liked MvL's coverage of vendor packages, in the PEP, since this could equally well apply to MSIs, python libraries distributed in bundles on OS X, debs, or RPMs. If this use-case were to be ignored, as one particular fellow seems to be advocating, then the broken packages and user confusion that has been going on for the last 5 years or so is just going to get worse. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 382: Namespace Packages
On 2009-04-07 19:46, P.J. Eby wrote: At 04:58 PM 4/7/2009 +0200, M.-A. Lemburg wrote: On 2009-04-07 16:05, P.J. Eby wrote: At 02:30 PM 4/7/2009 +0200, M.-A. Lemburg wrote: Wouldn't it be better to stick with a simpler approach and look for __pkg__.py files to detect namespace packages using that O(1) check ? Again - this wouldn't be O(1). More importantly, it breaks system packages, which now again have to deal with the conflicting file names if they want to install all portions into a single location. True, but since that means changing the package infrastructure, I think it's fair to ask distributors who want to use that approach to also take care of looking into the __pkg__.py files and merging them if necessary. Most of the time the __pkg__.py files will be empty, so that's not really much to ask for. This means your proposal actually doesn't add any benefit over the status quo, where you can have an __init__.py that does nothing but declare the package a namespace. We already have that now, and it doesn't need a new filename. Why would we expect OS vendors to start supporting it, just because we name it __pkg__.py instead of __init__.py? I lost you there. Since when do we support namespace packages in core Python without the need to add some form of magic support code to __init__.py ? My suggestion basically builds on the same idea as Martin's PEP, but uses a single __pkg__.py file as opposed to some non-Python file yaddayadda.pkg. Right... which completely obliterates the primary benefit of the original proposal compared to the status quo. That is, that the PEP 382 way is more compatible with system packaging tools. Without that benefit, there's zero gain in your proposal over having __init__.py files just call pkgutil.extend_path() (in the stdlib since 2.3, btw) or pkg_resources.declare_namespace() (similar functionality, but with zipfile support and some other niceties). IOW, your proposal doesn't actually improve the status quo in any way that I am able to determine, except that it calls for loading all the __pkg__.py modules, rather than just the first one. (And the setuptools implementation of namespace packages actually *does* load multiple __init__.py's, so that's still no change over the status quo for setuptools-using packages.) The purpose of the PEP is to create a standard for namespace packages. That's orthogonal to trying to enhance or change some existing techniques. I don't see the emphasis in the PEP on Linux distribution support and the remote possibility of them wanting to combine separate packages back into one package as good argument for adding yet another separate hierarchy of special files which Python scans during imports. That said, note that most distributions actually take the other route: they try to split up larger packages into smaller ones, so the argument becomes even weaker. It is much more important to standardize the approach than to try to extend some existing trickery and make them even more opaque than they already are by introducing yet another level of complexity. My alternative approach builds on existing methods and fits nicely with the __init__.py approach Python has already been using for more than a decade now. It's transparent, easy to understand and provides enough functionality to build upon - much like the original __init__.py idea. I've already laid out the arguments for and against it in my previous reply, so won't repeat them here. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Apr 14 2009) Python/Zope Consulting and Support ...http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ 2009-03-19: Released mxODBC.Connect 1.0.1 http://python.egenix.com/ ::: Try our new mxODBC.Connect Python Database Interface for free ! eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 382: Namespace Packages
At 05:02 PM 4/14/2009 +0200, M.-A. Lemburg wrote: I don't see the emphasis in the PEP on Linux distribution support and the remote possibility of them wanting to combine separate packages back into one package as good argument for adding yet another separate hierarchy of special files which Python scans during imports. That said, note that most distributions actually take the other route: they try to split up larger packages into smaller ones, so the argument becomes even weaker. I think you've misunderstood something about the use case. System packaging tools don't like separate packages to contain the *same file*. That means that they *can't* split a larger package up with your proposal, because every one of those packages would have to contain a __pkg__.py -- and thus be in conflict with each other. Either that, or they would have to make a separate system package containing *only* the __pkg__.py, and then make all packages using the namespace depend on it -- which is more work and requires greater co-ordination among packagers. Allowing each system package to contain its own .pkg or .nsp or whatever files, on the other hand, allows each system package to be built independently, without conflict between contents (i.e., having the same file), and without requiring a special pseudo-package to contain the additional file. Also, executing multiple __pkg__.py files means that when multiple system packages are installed to site-packages, only one of them could possibly be executed. (Note that, even though the system packages themselves are not combined, in practice they will all be installed to the same directory, i.e., site-packages or the platform equivalent thereof.) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 382: Namespace Packages
On 2009-04-14 18:27, P.J. Eby wrote: At 05:02 PM 4/14/2009 +0200, M.-A. Lemburg wrote: I don't see the emphasis in the PEP on Linux distribution support and the remote possibility of them wanting to combine separate packages back into one package as good argument for adding yet another separate hierarchy of special files which Python scans during imports. That said, note that most distributions actually take the other route: they try to split up larger packages into smaller ones, so the argument becomes even weaker. I think you've misunderstood something about the use case. System packaging tools don't like separate packages to contain the *same file*. That means that they *can't* split a larger package up with your proposal, because every one of those packages would have to contain a __pkg__.py -- and thus be in conflict with each other. Either that, or they would have to make a separate system package containing *only* the __pkg__.py, and then make all packages using the namespace depend on it -- which is more work and requires greater co-ordination among packagers. You are missing the point: When breaking up a large package that lives in site-packages into smaller distribution bundles, you don't need namespace packages at all, so the PEP doesn't apply. The way this works is by having a base distribution bundle that includes the needed __init__.py file and a set of extension bundles the add other files to the same directory (without including another copy of __init__.py). The extension bundles include a dependency on the base package to make sure that it always gets installed first. Debian has been using that approach for egenix-mx-base for years. Works great: http://packages.debian.org/source/lenny/egenix-mx-base eGenix has been using that approach for mx package add-ons as well - long before namespace packages where given that name :-) Please note that the PEP is about providing ways to have package parts live on sys.path that reintegrate themselves into a single package at import time. As such it's targeting Python developers that want to ship add-ons to existing packages, not Linux distributions (they usually have their own ideas about what goes where - something that's completely out-of- scope for the PEP). -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Apr 14 2009) Python/Zope Consulting and Support ...http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ 2009-03-19: Released mxODBC.Connect 1.0.1 http://python.egenix.com/ ::: Try our new mxODBC.Connect Python Database Interface for free ! eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 382: Namespace Packages
At 10:59 PM 4/14/2009 +0200, M.-A. Lemburg wrote: You are missing the point: When breaking up a large package that lives in site-packages into smaller distribution bundles, you don't need namespace packages at all, so the PEP doesn't apply. The way this works is by having a base distribution bundle that includes the needed __init__.py file and a set of extension bundles the add other files to the same directory (without including another copy of __init__.py). The extension bundles include a dependency on the base package to make sure that it always gets installed first. If we're going to keep that practice, there's no point to having the PEP: all three methods (base+extensions, pkgutil, setuptools) all work just fine as they are, with no changes to importing or the stdlib. In particular, without the feature of being able to drop that practice, there would be no reason for setuptools to adopt the PEP. That's why I'm -1 on your proposal: it's actually inferior to the methods we already have today. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 382: Namespace Packages
[Resent due to a python.org mail server problem] On 2009-04-03 22:07, Martin v. Löwis wrote: I'd like to extend the proposal to Python 2.7 and later. I don't object, but I also don't want to propose this, so I added it to the discussion. My (and perhaps other people's) concern is that 2.7 might well be the last release of the 2.x series. If so, adding this feature to it would make 2.7 an odd special case for users and providers of third party tools. I certainly hope that we'll see more useful features backported from 3.x to the 2.x series or forward ported from 2.x to 3.x (depending on what the core developer preferences are). Regarding this particular PEP, it is well possible to implement an importer that provides the functionality for Python 2.3-2.7 versions, so it doesn't have to be an odd special case. That's going to slow down Python package detection a lot - you'd replace an O(1) test with an O(n) scan. I question that claim. In traditional Unix systems, the file system driver performs a linear search of the directory, so it's rather O(n)-in-kernel vs. O(n)-in-Python. Even for advanced file systems, you need at least O(log n) to determine whether a specific file is in a directory. For all practical purposes, the package directory will fit in a single disk block (containing a single .pkg file, and one or few subpackages), making listdir complete as fast as stat. On second thought, you're right, it won't be that costly. It requires an os.listdir() scan due to the wildcard approach and in some cases, such a scan may not be possible, e.g. when using frozen packages. Indeed, the freeze mechanism would not even add the .pkg files - it only handles .py file content. The same is true for distutils, MANIFEST generators and other installer mechanisms - it would have to learn to package the .pkg files along with the Python files. Another problem with the .pkg file approach is that the file extension is already in use for e.g. Mac OS X installers. You don't have those issues with the __pkg__.py file approach I suggested. Wouldn't it be better to stick with a simpler approach and look for __pkg__.py files to detect namespace packages using that O(1) check ? Again - this wouldn't be O(1). More importantly, it breaks system packages, which now again have to deal with the conflicting file names if they want to install all portions into a single location. True, but since that means changing the package infrastructure, I think it's fair to ask distributors who want to use that approach to also take care of looking into the __pkg__.py files and merging them if necessary. Most of the time the __pkg__.py files will be empty, so that's not really much to ask for. This would also avoid any issues you'd otherwise run into if you want to maintain this scheme in an importer that doesn't have access to a list of files in a package directory, but is well capable for the checking the existence of a file. Do you have a specific mechanism in mind? Yes: frozen modules and imports straight from a web resource. The .pkg file approach requires a directory scan and additional support from all importers. The __pkg__.py approach I suggested can use existing importers without modifications by checking for the existence of such a Python module in an importer managed resource. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Apr 07 2009) Python/Zope Consulting and Support ...http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ 2009-03-19: Released mxODBC.Connect 1.0.1 http://python.egenix.com/ ::: Try our new mxODBC.Connect Python Database Interface for free ! eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 382: Namespace Packages
On 2009-04-03 02:44, P.J. Eby wrote: At 10:33 PM 4/2/2009 +0200, M.-A. Lemburg wrote: Alternative Approach: - Wouldn't it be better to stick with a simpler approach and look for __pkg__.py files to detect namespace packages using that O(1) check ? One of the namespace packages, the defining namespace package, will have to include a __init__.py file. Note that there is no such thing as a defining namespace package -- namespace package contents are symmetrical peers. That was a definition :-) Definition namespace package := the namespace package having the __pkg__.py file This is useful to have since packages allowing integration of other sub-packages typically come as a base package with some basic infra-structure in place which is required by all other namespace packages. If the __init__.py file is not found among the namespace directories, the importer will have to raise an exception, since the result would not be a proper Python package. * It's possible to have a defining package dir and add-one package dirs. Also possible in the PEP, although the __init__.py must be in the first such directory on sys.path. (However, such defining packages are not that common now, due to tool limitations.) That's a strange limitation of the PEP. Why should the location of the __init__.py file depend on the order of sys.path ? -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Apr 03 2009) Python/Zope Consulting and Support ...http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ 2009-03-19: Released mxODBC.Connect 1.0.1 http://python.egenix.com/ ::: Try our new mxODBC.Connect Python Database Interface for free ! eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 382: Namespace Packages
At 02:30 PM 4/7/2009 +0200, M.-A. Lemburg wrote: Wouldn't it be better to stick with a simpler approach and look for __pkg__.py files to detect namespace packages using that O(1) check ? Again - this wouldn't be O(1). More importantly, it breaks system packages, which now again have to deal with the conflicting file names if they want to install all portions into a single location. True, but since that means changing the package infrastructure, I think it's fair to ask distributors who want to use that approach to also take care of looking into the __pkg__.py files and merging them if necessary. Most of the time the __pkg__.py files will be empty, so that's not really much to ask for. This means your proposal actually doesn't add any benefit over the status quo, where you can have an __init__.py that does nothing but declare the package a namespace. We already have that now, and it doesn't need a new filename. Why would we expect OS vendors to start supporting it, just because we name it __pkg__.py instead of __init__.py? ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 382: Namespace Packages
On 2009-04-07 16:05, P.J. Eby wrote: At 02:30 PM 4/7/2009 +0200, M.-A. Lemburg wrote: Wouldn't it be better to stick with a simpler approach and look for __pkg__.py files to detect namespace packages using that O(1) check ? Again - this wouldn't be O(1). More importantly, it breaks system packages, which now again have to deal with the conflicting file names if they want to install all portions into a single location. True, but since that means changing the package infrastructure, I think it's fair to ask distributors who want to use that approach to also take care of looking into the __pkg__.py files and merging them if necessary. Most of the time the __pkg__.py files will be empty, so that's not really much to ask for. This means your proposal actually doesn't add any benefit over the status quo, where you can have an __init__.py that does nothing but declare the package a namespace. We already have that now, and it doesn't need a new filename. Why would we expect OS vendors to start supporting it, just because we name it __pkg__.py instead of __init__.py? I lost you there. Since when do we support namespace packages in core Python without the need to add some form of magic support code to __init__.py ? My suggestion basically builds on the same idea as Martin's PEP, but uses a single __pkg__.py file as opposed to some non-Python file yaddayadda.pkg. Here's a copy of the proposal, with some additional discussion bullets added: Alternative Approach: - Wouldn't it be better to stick with a simpler approach and look for __pkg__.py files to detect namespace packages using that O(1) check ? This would also avoid any issues you'd otherwise run into if you want to maintain this scheme in an importer that doesn't have access to a list of files in a package directory, but is well capable for the checking the existence of a file. Mechanism: -- If the import mechanism finds a matching namespace package (a directory with a __pkg__.py file), it then goes into namespace package scan mode and scans the complete sys.path for more occurrences of the same namespace package. The import loads all __pkg__.py files of matching namespace packages having the same package name during the search. One of the namespace packages, the defining namespace package, will have to include a __init__.py file. After having scanned all matching namespace packages and loading the __pkg__.py files in the order of the search, the import mechanism then sets the packages .__path__ attribute to include all namespace package directories found on sys.path and finally executes the __init__.py file. (Please let me know if the above is not clear, I will then try to follow up on it.) Discussion: --- The above mechanism allows the same kind of flexibility we already have with the existing normal __init__.py mechanism. * It doesn't add yet another .pth-style sys.path extension (which are difficult to manage in installations). * It always uses the same naive sys.path search strategy. The strategy is not determined by some file contents. * The search is only done once - on the first import of the package. * It's possible to have a defining package dir and add-one package dirs. * The search does not depend on the order of directories in sys.path. There's no requirement for the defining package to appear first on sys.path. * Namespace packages are easy to recognize by testing for a single resource. * There's no conflict with existing files using the .pkg extension such as Mac OS X installer files or Solaris packages. * Namespace __pkg__.py modules can provide extra meta-information, logging, etc. to simplify debugging namespace package setups. * It's possible to freeze such setups, to put them into ZIP files, or only have parts of it in a ZIP file and the other parts in the file-system. * There's no need for a package directory scan, allowing the mechanism to also work with resources that do not permit to (easily and efficiently) scan the contents of a package directory, e.g. frozen packages or imports from web resources. Caveats: * Changes to sys.path will not result in an automatic rescan for additional namespace packages, if the package was already loaded. However, we could have a function to make such a rescan explicit. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Apr 07 2009) Python/Zope Consulting and Support ...http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ 2009-03-19: Released mxODBC.Connect 1.0.1 http://python.egenix.com/ ::: Try our new mxODBC.Connect Python Database Interface for free ! eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO
Re: [Python-Dev] PEP 382: Namespace Packages
On Tue, Apr 7, 2009 at 11:58 PM, M.-A. Lemburg m...@egenix.com wrote: This means your proposal actually doesn't add any benefit over the status quo, where you can have an __init__.py that does nothing but declare the package a namespace. We already have that now, and it doesn't need a new filename. Why would we expect OS vendors to start supporting it, just because we name it __pkg__.py instead of __init__.py? I lost you there. Since when do we support namespace packages in core Python without the need to add some form of magic support code to __init__.py ? I think P. Eby refers to the problem that most packaging systems don't like several packages to have the same file - be it empty or not. That's my main personal grip against namespace packages, and from this POV, I think it is fair to say the proposal does not solve anything. Not that I have a solution, of course :) cheers, David My suggestion basically builds on the same idea as Martin's PEP, but uses a single __pkg__.py file as opposed to some non-Python file yaddayadda.pkg. Here's a copy of the proposal, with some additional discussion bullets added: Alternative Approach: - Wouldn't it be better to stick with a simpler approach and look for __pkg__.py files to detect namespace packages using that O(1) check ? This would also avoid any issues you'd otherwise run into if you want to maintain this scheme in an importer that doesn't have access to a list of files in a package directory, but is well capable for the checking the existence of a file. Mechanism: -- If the import mechanism finds a matching namespace package (a directory with a __pkg__.py file), it then goes into namespace package scan mode and scans the complete sys.path for more occurrences of the same namespace package. The import loads all __pkg__.py files of matching namespace packages having the same package name during the search. One of the namespace packages, the defining namespace package, will have to include a __init__.py file. After having scanned all matching namespace packages and loading the __pkg__.py files in the order of the search, the import mechanism then sets the packages .__path__ attribute to include all namespace package directories found on sys.path and finally executes the __init__.py file. (Please let me know if the above is not clear, I will then try to follow up on it.) Discussion: --- The above mechanism allows the same kind of flexibility we already have with the existing normal __init__.py mechanism. * It doesn't add yet another .pth-style sys.path extension (which are difficult to manage in installations). * It always uses the same naive sys.path search strategy. The strategy is not determined by some file contents. * The search is only done once - on the first import of the package. * It's possible to have a defining package dir and add-one package dirs. * The search does not depend on the order of directories in sys.path. There's no requirement for the defining package to appear first on sys.path. * Namespace packages are easy to recognize by testing for a single resource. * There's no conflict with existing files using the .pkg extension such as Mac OS X installer files or Solaris packages. * Namespace __pkg__.py modules can provide extra meta-information, logging, etc. to simplify debugging namespace package setups. * It's possible to freeze such setups, to put them into ZIP files, or only have parts of it in a ZIP file and the other parts in the file-system. * There's no need for a package directory scan, allowing the mechanism to also work with resources that do not permit to (easily and efficiently) scan the contents of a package directory, e.g. frozen packages or imports from web resources. Caveats: * Changes to sys.path will not result in an automatic rescan for additional namespace packages, if the package was already loaded. However, we could have a function to make such a rescan explicit. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Apr 07 2009) Python/Zope Consulting and Support ... http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ 2009-03-19: Released mxODBC.Connect 1.0.1 http://python.egenix.com/ ::: Try our new mxODBC.Connect Python Database Interface for free ! eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ ___ Python-Dev mailing list Python-Dev@python.org
Re: [Python-Dev] PEP 382: Namespace Packages
At 04:58 PM 4/7/2009 +0200, M.-A. Lemburg wrote: On 2009-04-07 16:05, P.J. Eby wrote: At 02:30 PM 4/7/2009 +0200, M.-A. Lemburg wrote: Wouldn't it be better to stick with a simpler approach and look for __pkg__.py files to detect namespace packages using that O(1) check ? Again - this wouldn't be O(1). More importantly, it breaks system packages, which now again have to deal with the conflicting file names if they want to install all portions into a single location. True, but since that means changing the package infrastructure, I think it's fair to ask distributors who want to use that approach to also take care of looking into the __pkg__.py files and merging them if necessary. Most of the time the __pkg__.py files will be empty, so that's not really much to ask for. This means your proposal actually doesn't add any benefit over the status quo, where you can have an __init__.py that does nothing but declare the package a namespace. We already have that now, and it doesn't need a new filename. Why would we expect OS vendors to start supporting it, just because we name it __pkg__.py instead of __init__.py? I lost you there. Since when do we support namespace packages in core Python without the need to add some form of magic support code to __init__.py ? My suggestion basically builds on the same idea as Martin's PEP, but uses a single __pkg__.py file as opposed to some non-Python file yaddayadda.pkg. Right... which completely obliterates the primary benefit of the original proposal compared to the status quo. That is, that the PEP 382 way is more compatible with system packaging tools. Without that benefit, there's zero gain in your proposal over having __init__.py files just call pkgutil.extend_path() (in the stdlib since 2.3, btw) or pkg_resources.declare_namespace() (similar functionality, but with zipfile support and some other niceties). IOW, your proposal doesn't actually improve the status quo in any way that I am able to determine, except that it calls for loading all the __pkg__.py modules, rather than just the first one. (And the setuptools implementation of namespace packages actually *does* load multiple __init__.py's, so that's still no change over the status quo for setuptools-using packages.) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 382: Namespace Packages
Martin v. Löwis wrote: Chris Withers wrote: Martin v. Löwis wrote: I propose the following PEP for inclusion to Python 3.1. Please comment. Would this support the following case: I have a package called mortar, which defines useful stuff: from mortar import content, ... I now want to distribute large optional chunks separately, but ideally so that the following will will work: from mortar.rbd import ... from mortar.zodb import ... from mortar.wsgi import ... Does the PEP support this? That's the primary purpose of the PEP. Are you sure? Does the pep really allow for: from mortar import content from mortar.rdb import something ...where 'content' is a function defined in mortar/__init__.py and 'something' is a function defined in mortar/rdb/__init__.py *and* the following are separate distributions on PyPI: - mortar - mortar.rdb ...where 'mortar' does not contain 'mortar.rdb'. You can do this today already (see the zope package, No, they have nothing but a (functionally) empty __init__.py in the zope package. cheers, Chris -- Simplistix - Content Management, Zope Python Consulting - http://www.simplistix.co.uk ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 382: Namespace Packages
On Thu, Apr 2, 2009 at 4:33 PM, M.-A. Lemburg m...@egenix.com wrote: On 2009-04-02 17:32, Martin v. Löwis wrote: I propose the following PEP for inclusion to Python 3.1. Thanks for picking this up. I'd like to extend the proposal to Python 2.7 and later. -1 to adding it to the 2.x series. There was much discussion around adding features to 2.x *and* 3.0, and the consensus seemed to *not* add new features to 2.x and use those new features as carrots to help lead people into 3.0. jesse ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 382: Namespace Packages
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Apr 6, 2009, at 9:21 AM, Jesse Noller wrote: On Thu, Apr 2, 2009 at 4:33 PM, M.-A. Lemburg m...@egenix.com wrote: On 2009-04-02 17:32, Martin v. Löwis wrote: I propose the following PEP for inclusion to Python 3.1. Thanks for picking this up. I'd like to extend the proposal to Python 2.7 and later. -1 to adding it to the 2.x series. There was much discussion around adding features to 2.x *and* 3.0, and the consensus seemed to *not* add new features to 2.x and use those new features as carrots to help lead people into 3.0. Actually, isn't the policy just that nothing can go into 2.7 that isn't backported from 3.1? Whether the actual backport happens or not is up to the developer though. OTOH, we talked about a lot of things and my recollection is probably fuzzy. Barry -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (Darwin) iQCVAwUBSdoDAXEjvBPtnXfVAQIrPgQAse7BXQfPYHJJ/g3HNEtc0UmZZ9MCNtGc sIoZ2EHRVz+pylZT9fmSmorJdIdFvAj7E43tKsV2bQpo/am9XlL10SMn3k0KLxnF vNCi39nB1B7Uktbnrlpnfo4u93suuEqYexEwrkDhJuTMeye0Cxg0os5aysryuPza mKr5jsqkV5c= =Y9iP -END PGP SIGNATURE- ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 382: Namespace Packages
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Apr 6, 2009, at 9:21 AM, Jesse Noller wrote: On Thu, Apr 2, 2009 at 4:33 PM, M.-A. Lemburg m...@egenix.com wrote: On 2009-04-02 17:32, Martin v. Löwis wrote: I propose the following PEP for inclusion to Python 3.1. Thanks for picking this up. I'd like to extend the proposal to Python 2.7 and later. -1 to adding it to the 2.x series. There was much discussion around adding features to 2.x *and* 3.0, and the consensus seemed to *not* add new features to 2.x and use those new features as carrots to help lead people into 3.0. Actually, isn't the policy just that nothing can go into 2.7 that isn't backported from 3.1? Whether the actual backport happens or not is up to the developer though. OTOH, we talked about a lot of things and my recollection is probably fuzzy. I believe Barry is correct. The official policy is no features in 2.7 that aren't also in 3.1. I personally think I'm not going to put anything else in 2.7, specifically the ',' formatter stuff from PEP 378. 3.1 has diverged too far from 2.7 in this regard to make the backport easy to do. But this decision is left up to the individual committer. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 382: Namespace Packages
At 02:00 PM 4/6/2009 +0100, Chris Withers wrote: Martin v. Löwis wrote: Chris Withers wrote: Would this support the following case: I have a package called mortar, which defines useful stuff: from mortar import content, ... I now want to distribute large optional chunks separately, but ideally so that the following will will work: from mortar.rbd import ... from mortar.zodb import ... from mortar.wsgi import ... Does the PEP support this? That's the primary purpose of the PEP. Are you sure? Does the pep really allow for: from mortar import content from mortar.rdb import something ...where 'content' is a function defined in mortar/__init__.py and 'something' is a function defined in mortar/rdb/__init__.py *and* the following are separate distributions on PyPI: - mortar - mortar.rdb ...where 'mortar' does not contain 'mortar.rdb'. See the third paragraph of http://www.python.org/dev/peps/pep-0382/#discussion ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 382: Namespace Packages
P.J. Eby wrote: See the third paragraph of http://www.python.org/dev/peps/pep-0382/#discussion Indeed, I guess the PEP could be made more explanatory then 'cos, as a packager, I don't see what I'd put in the various setup.py and __init__.py to make this work... That said, I'm delighted to hear it's going to be possible and wholeheartedly support the PEP and it's backporting to 2.7 as a result... cheers, Chris -- Simplistix - Content Management, Zope Python Consulting - http://www.simplistix.co.uk ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 382: Namespace Packages
On Mon, Apr 6, 2009 at 9:26 AM, Barry Warsaw ba...@python.org wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Apr 6, 2009, at 9:21 AM, Jesse Noller wrote: On Thu, Apr 2, 2009 at 4:33 PM, M.-A. Lemburg m...@egenix.com wrote: On 2009-04-02 17:32, Martin v. Löwis wrote: I propose the following PEP for inclusion to Python 3.1. Thanks for picking this up. I'd like to extend the proposal to Python 2.7 and later. -1 to adding it to the 2.x series. There was much discussion around adding features to 2.x *and* 3.0, and the consensus seemed to *not* add new features to 2.x and use those new features as carrots to help lead people into 3.0. Actually, isn't the policy just that nothing can go into 2.7 that isn't backported from 3.1? Whether the actual backport happens or not is up to the developer though. OTOH, we talked about a lot of things and my recollection is probably fuzzy. Barry That *is* the official policy, but there was discussions around no further backporting of features from 3.1 into 2.x, therefore providing more of an upgrade incentive ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 382: Namespace Packages
On Mon, 6 Apr 2009 at 12:00, Jesse Noller wrote: On Mon, Apr 6, 2009 at 9:26 AM, Barry Warsaw ba...@python.org wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Apr 6, 2009, at 9:21 AM, Jesse Noller wrote: On Thu, Apr 2, 2009 at 4:33 PM, M.-A. Lemburg m...@egenix.com wrote: On 2009-04-02 17:32, Martin v. L?wis wrote: I propose the following PEP for inclusion to Python 3.1. Thanks for picking this up. I'd like to extend the proposal to Python 2.7 and later. -1 to adding it to the 2.x series. There was much discussion around adding features to 2.x *and* 3.0, and the consensus seemed to *not* add new features to 2.x and use those new features as carrots to help lead people into 3.0. Actually, isn't the policy just that nothing can go into 2.7 that isn't backported from 3.1? ?Whether the actual backport happens or not is up to the developer though. ?OTOH, we talked about a lot of things and my recollection is probably fuzzy. Barry That *is* the official policy, but there was discussions around no further backporting of features from 3.1 into 2.x, therefore providing more of an upgrade incentive My sense was that this wasn't proposed as a hard and fast rule, more as a strongly suggested guideline. And in this case, I think you could argue that the PEP is actually fixing a bug in the current namespace packaging system. Some projects, especially the large ones where this matters most, are going to have to maintain backward compatibility for 2.x for a long time even as 3.x adoption accelerates. It seems a shame to require packagers to continue to deal with the problems caused by the current system even after all the platforms have made it to 2.7+. --David___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 382: Namespace Packages
On Mon, Apr 6, 2009 at 12:28 PM, R. David Murray rdmur...@bitdance.com wrote: On Mon, 6 Apr 2009 at 12:00, Jesse Noller wrote: On Mon, Apr 6, 2009 at 9:26 AM, Barry Warsaw ba...@python.org wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Apr 6, 2009, at 9:21 AM, Jesse Noller wrote: On Thu, Apr 2, 2009 at 4:33 PM, M.-A. Lemburg m...@egenix.com wrote: On 2009-04-02 17:32, Martin v. Löwis wrote: I propose the following PEP for inclusion to Python 3.1. Thanks for picking this up. I'd like to extend the proposal to Python 2.7 and later. -1 to adding it to the 2.x series. There was much discussion around adding features to 2.x *and* 3.0, and the consensus seemed to *not* add new features to 2.x and use those new features as carrots to help lead people into 3.0. Actually, isn't the policy just that nothing can go into 2.7 that isn't backported from 3.1? Whether the actual backport happens or not is up to the developer though. OTOH, we talked about a lot of things and my recollection is probably fuzzy. Barry That *is* the official policy, but there was discussions around no further backporting of features from 3.1 into 2.x, therefore providing more of an upgrade incentive My sense was that this wasn't proposed as a hard and fast rule, more as a strongly suggested guideline. And in this case, I think you could argue that the PEP is actually fixing a bug in the current namespace packaging system. Some projects, especially the large ones where this matters most, are going to have to maintain backward compatibility for 2.x for a long time even as 3.x adoption accelerates. It seems a shame to require packagers to continue to deal with the problems caused by the current system even after all the platforms have made it to 2.7+. --David I know it wasn't a hard and fast rule; also, with 3to2 already being worked on, the barrier of maintenance and back porting is going to be lowered. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 382: Namespace Packages
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Jesse Noller wrote: On Mon, Apr 6, 2009 at 12:28 PM, R. David Murray rdmur...@bitdance.com wrote: On Mon, 6 Apr 2009 at 12:00, Jesse Noller wrote: On Mon, Apr 6, 2009 at 9:26 AM, Barry Warsaw ba...@python.org wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Apr 6, 2009, at 9:21 AM, Jesse Noller wrote: On Thu, Apr 2, 2009 at 4:33 PM, M.-A. Lemburg m...@egenix.com wrote: On 2009-04-02 17:32, Martin v. Löwis wrote: I propose the following PEP for inclusion to Python 3.1. Thanks for picking this up. I'd like to extend the proposal to Python 2.7 and later. -1 to adding it to the 2.x series. There was much discussion around adding features to 2.x *and* 3.0, and the consensus seemed to *not* add new features to 2.x and use those new features as carrots to help lead people into 3.0. Actually, isn't the policy just that nothing can go into 2.7 that isn't backported from 3.1? Whether the actual backport happens or not is up to the developer though. OTOH, we talked about a lot of things and my recollection is probably fuzzy. Barry That *is* the official policy, but there was discussions around no further backporting of features from 3.1 into 2.x, therefore providing more of an upgrade incentive My sense was that this wasn't proposed as a hard and fast rule, more as a strongly suggested guideline. And in this case, I think you could argue that the PEP is actually fixing a bug in the current namespace packaging system. Some projects, especially the large ones where this matters most, are going to have to maintain backward compatibility for 2.x for a long time even as 3.x adoption accelerates. It seems a shame to require packagers to continue to deal with the problems caused by the current system even after all the platforms have made it to 2.7+. --David I know it wasn't a hard and fast rule; also, with 3to2 already being worked on, the barrier of maintenance and back porting is going to be lowered. My understanding from the summit is that the only point in a 2.7 release at all is to lower the speed bumps which make porting from 2.x to 3.x hard for large codebases. In this case, having a consistent spelling for namespace packages between 2.7 and 3.1 would incent those applications / frameworks / libraries to move to 2.7, and therefore ease getting them to 3.1. Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFJ2jaR+gerLs4ltQ4RAsi1AJ0cJyKsoP5SlOcBlnzLr6MB11ZoNwCg1Kil 4O2M0sZG+jH12s22p2AmXWk= =DLRM -END PGP SIGNATURE- ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 382: Namespace Packages
Perhaps we could add something like a sys.namespace_packages that would be updated by this mechanism? Then, pkg_resources could check both that and its internal registry to be both backward and forward compatible. I could see no problem with that, so I have added this to the PEP. Thanks for the feedback, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 382: Namespace Packages
Chris Withers wrote: Martin v. Löwis wrote: I propose the following PEP for inclusion to Python 3.1. Please comment. Would this support the following case: I have a package called mortar, which defines useful stuff: from mortar import content, ... I now want to distribute large optional chunks separately, but ideally so that the following will will work: from mortar.rbd import ... from mortar.zodb import ... from mortar.wsgi import ... Does the PEP support this? That's the primary purpose of the PEP. You can do this today already (see the zope package, and the reference to current techniques in the PEP), but the PEP provides a cleaner way. In each chunk (which the PEP calls portion), you had a structure like this: mortar/ mortar/rbd.pkg (contains just *) mortar/rbd.py or mortar/ mortar/zobd.pkg mortar/zobd/ mortar/zobd/__init__.py mortar/zobd/backends.py As a site effect, you can also do import mortar, but that would just give you the (nearly) empty namespace package, whose only significant contents is the variable __path__. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 382: Namespace Packages
On 08:15 pm, mar...@v.loewis.de wrote: Note that there is no such thing as a defining namespace package -- namespace package contents are symmetrical peers. With the PEP, a defining package becomes possible - at most one portion can define an __init__.py. For what it's worth, this is a _super_ useful feature for Twisted. We have one defining package for the twisted package (twisted core) and then a bunch of other things which want to put things into twisted.* (twisted.web, twisted.conch, et. al.). For debian we already have separate packages, but such a definition of namespace packages would allow us to actually have things separated out on the cheeseshop as well. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 382: Namespace Packages
At 10:15 PM 4/3/2009 +0200, Martin v. Löwis wrote:
I should make it clear that this is not the case. I envision it to work
this way: import zope
- searches sys.path, until finding either a directory zope, or a file
zope.{py,pyc,pyd,...}
- if it is a directory, it checks for .pkg files. If it finds any,
it processes them, extending __path__.
- it *then* checks for __init__.py, taking the first hit anywhere
on __path__ (just like any module import would)
- if no .pkg was found, nor an __init__.py, it proceeds with the next
sys.path item (skipping the directory entirely)
Ah, I missed that. Maybe the above should be added to the PEP to clarify.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 382: Namespace Packages
On Fri, Apr 3, 2009 at 13:15, Martin v. Löwis mar...@v.loewis.de wrote:
Note that there is no such thing as a defining namespace package --
namespace package contents are symmetrical peers.
With the PEP, a defining package becomes possible - at most one
portion can define an __init__.py.
I know that the current mechanisms don't support it, and it might
not be useful in general, but now there is a clean way of doing it,
so I wouldn't exclude it. Distribution-wise, all distributions
relying on the defining package would need to require (or
install_require, or depend on) it.
The above are also true for using only a '*' in .pkg files -- in that
event there are no sys.path changes. (Frankly, I'm doubtful that
anybody is using extend_path and .pkg files to begin with, so I'd be
fine with a proposal that instead used something like '.nsp' files that
didn't even need to be opened and read -- which would let the directory
scan stop at the first .nsp file found.
That would work for me as well. Nobody at PyCon could remember where
.pkg files came from.
I believe the PEP does this as well, IIUC.
Correct.
* It's possible to have a defining package dir and add-one package
dirs.
Also possible in the PEP, although the __init__.py must be in the first
such directory on sys.path.
I should make it clear that this is not the case. I envision it to work
this way: import zope
- searches sys.path, until finding either a directory zope, or a file
zope.{py,pyc,pyd,...}
- if it is a directory, it checks for .pkg files. If it finds any,
it processes them, extending __path__.
- it *then* checks for __init__.py, taking the first hit anywhere
on __path__ (just like any module import would)
Just so people know how this __init__ search could be done such that
__path__ is set from the .pkg is to treat it as a reload (assuming .pkg
files can only be found off of sys.path).
-Brett
- if no .pkg was found, nor an __init__.py, it proceeds with the next
sys.path item (skipping the directory entirely)
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 382: Namespace Packages
At 10:32 AM 4/2/2009 -0500, Martin v. Löwis wrote: I propose the following PEP for inclusion to Python 3.1. Please comment. An excellent idea. One thing I am not 100% clear on, is how to get additions to sys.path to work correctly with this. Currently, when pkg_resources adds a new egg to sys.path, it uses its existing registry of namespace packages in order to locate which packages need __path__ fixups. It seems under this proposal that it would have to scan sys.modules for objects with __path__ attributes that are lists that begin with a '*', instead... which is a bit troubling because sys.modules doesn't always only contain module objects. Many major frameworks place lazy module objects, and module proxies or wrappers of various sorts in there, so scanning through it arbitrarily is not really a good idea. Perhaps we could add something like a sys.namespace_packages that would be updated by this mechanism? Then, pkg_resources could check both that and its internal registry to be both backward and forward compatible. Apart from that, this mechanism sounds great! I only wish there was a way to backport it all the way to 2.3 so I could drop the messy bits from setuptools. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 382: Namespace Packages
P.J. Eby wrote: Apart from that, this mechanism sounds great! I only wish there was a way to backport it all the way to 2.3 so I could drop the messy bits from setuptools. Maybe we could? :-) Chris -- Simplistix - Content Management, Zope Python Consulting - http://www.simplistix.co.uk ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 382: Namespace Packages
On 2009-04-02 17:32, Martin v. Löwis wrote: I propose the following PEP for inclusion to Python 3.1. Thanks for picking this up. I'd like to extend the proposal to Python 2.7 and later. Please comment. Regards, Martin Specification = Rather than using an imperative mechanism for importing packages, a declarative approach is proposed here, as an extension to the existing ``*.pkg`` mechanism. The import statement is extended so that it directly considers ``*.pkg`` files during import; a directory is considered a package if it either contains a file named __init__.py, or a file whose name ends with .pkg. That's going to slow down Python package detection a lot - you'd replace an O(1) test with an O(n) scan. Alternative Approach: - Wouldn't it be better to stick with a simpler approach and look for __pkg__.py files to detect namespace packages using that O(1) check ? This would also avoid any issues you'd otherwise run into if you want to maintain this scheme in an importer that doesn't have access to a list of files in a package directory, but is well capable for the checking the existence of a file. Mechanism: -- If the import mechanism finds a matching namespace package (a directory with a __pkg__.py file), it then goes into namespace package scan mode and scans the complete sys.path for more occurrences of the same namespace package. The import loads all __pkg__.py files of matching namespace packages having the same package name during the search. One of the namespace packages, the defining namespace package, will have to include a __init__.py file. After having scanned all matching namespace packages and loading the __pkg__.py files in the order of the search, the import mechanism then sets the packages .__path__ attribute to include all namespace package directories found on sys.path and finally executes the __init__.py file. (Please let me know if the above is not clear, I will then try to follow up on it.) Discussion: --- The above mechanism allows the same kind of flexibility we already have with the existing normal __init__.py mechanism. * It doesn't add yet another .pth-style sys.path extension (which are difficult to manage in installations). * It always uses the same naive sys.path search strategy. The strategy is not determined by some file contents. * The search is only done once - on the first import of the package. * It's possible to have a defining package dir and add-one package dirs. * Namespace packages are easy to recognize by testing for a single resource. * Namespace __pkg__.py modules can provide extra meta-information, logging, etc. to simplify debugging namespace package setups. * It's possible to freeze such setups, to put them into ZIP files, or only have parts of it in a ZIP file and the other parts in the file-system. Caveats: * Changes to sys.path will not result in an automatic rescan for additional namespace packages, if the package was already loaded. However, we could have a function to make such a rescan explicit. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Apr 02 2009) Python/Zope Consulting and Support ...http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ 2009-03-19: Released mxODBC.Connect 1.0.1 http://python.egenix.com/ ::: Try our new mxODBC.Connect Python Database Interface for free ! eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 382: Namespace Packages
At 10:33 PM 4/2/2009 +0200, M.-A. Lemburg wrote: That's going to slow down Python package detection a lot - you'd replace an O(1) test with an O(n) scan. I thought about this too, but it's pretty trivial considering that the only time it takes effect is when you have a directory name that matches the name you're importing, and that it will only happen once for that directory, unless there is no package on sys.path with that name, and the program tries to import the package multiple times. In other words, the overhead isn't likely to be much, compared to the time needed to say, open and marshal even a trivial __init__.py file. Alternative Approach: - Wouldn't it be better to stick with a simpler approach and look for __pkg__.py files to detect namespace packages using that O(1) check ? I thought the same thing (or more precisely, a single .pkg file), but when I got lower in the PEP I saw the reason was to support system packages not having overlapping filenames. The PEP could probably be a little clearer about the connection between needing *.pkg and the system-package use case. One of the namespace packages, the defining namespace package, will have to include a __init__.py file. Note that there is no such thing as a defining namespace package -- namespace package contents are symmetrical peers. The above mechanism allows the same kind of flexibility we already have with the existing normal __init__.py mechanism. * It doesn't add yet another .pth-style sys.path extension (which are difficult to manage in installations). * It always uses the same naive sys.path search strategy. The strategy is not determined by some file contents. The above are also true for using only a '*' in .pkg files -- in that event there are no sys.path changes. (Frankly, I'm doubtful that anybody is using extend_path and .pkg files to begin with, so I'd be fine with a proposal that instead used something like '.nsp' files that didn't even need to be opened and read -- which would let the directory scan stop at the first .nsp file found. * The search is only done once - on the first import of the package. I believe the PEP does this as well, IIUC. * It's possible to have a defining package dir and add-one package dirs. Also possible in the PEP, although the __init__.py must be in the first such directory on sys.path. (However, such defining packages are not that common now, due to tool limitations.) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 382: Namespace Packages
Martin v. Löwis schrieb: I propose the following PEP for inclusion to Python 3.1. Please comment. Regards, Martin Abstract Namespace packages are a mechanism for splitting a single Python package across multiple directories on disk. In current Python versions, an algorithm to compute the packages __path__ must be formulated. With the enhancement proposed here, the import machinery itself will construct the list of directories that make up the package. +1 speaking as a downstream packaging python for Debian/Ubuntu I welcome this approach. The current practice of shipping the very same file (__init__.py) in different packages leads to conflicts for the installation of these packages (this is not specific to dpkg, but is true for rpm packaging as well). Current practice of packaging (for downstreams) so called name space packages is: - either to split out the namespace __init__.py into a separate (linux distribution) package (needing manual packaging effort for each name space package) - using downstream specific packaging techniques to handle conflicting files (diversions) - replicating the current behaviour of setuptools simply overwriting the file conflicts. Following this proposal (downstream) packaging of namespace packages is made possible independent of any manual downstream packaging decisions or any downstream specific packaging decisions. Matthias ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 382: Namespace Packages
At 03:21 AM 4/3/2009 +0200, Matthias Klose wrote: +1 speaking as a downstream packaging python for Debian/Ubuntu I welcome this approach. The current practice of shipping the very same file (__init__.py) in different packages leads to conflicts for the installation of these packages (this is not specific to dpkg, but is true for rpm packaging as well). Current practice of packaging (for downstreams) so called name space packages is: - either to split out the namespace __init__.py into a separate(linux distribution) package (needing manual packaging effort for eachname space package) - using downstream specific packaging techniques to handle conflicting files(diversions) - replicating the current behaviour of setuptools simply overwriting thefile conflicts. Following this proposal (downstream) packaging of namespace packages is made possible independent of any manual downstream packaging decisions or any downstream specific packaging decisions A clarification: setuptools does not currently install the __init__.py file when installing in --single-version-externally-managed or --root mode. Instead, it uses a project-version-nspkg.pth file that essentially simulates a variation of Martin's .pkg proposal, by abusing .pth file support. If this PEP is adopted, setuptools would replace its nspkg.pth file with a .pkg file on Python versions that provide native support for .pkg imports, keeping the .pth file only for older Pythons. (.egg files and directories will not be affected by the change, unless the zipimport module will also supports .pkg files... and again, only for Python versions that support the new approach.) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
