Re: handling /usr/local/lib/python2.x/site-packages in sys.path
On Tue, Mar 11, 2008 at 08:43:06AM +0100, Matthias Klose wrote:
Currently Debian's python has /usr/local/lib/python2.x/site-packages
in sys.path allowing for installation of local modules. Barry did
point out that this conflicts with a python installation, where
/usr/local is the default prefix, and the system python gets modules
from the local installation. Some suggestions were made to fix this:
[...]
- add another path (e.g. /usr/local/python/lib2.x/site-packages),
and remove the /usr/local/lib/python2.x/site-packages path after
the next release. Does provide an upgrade path, but doesn't solve
the probem immediately.
Discussion on distutils-sig[1] seems to point out that no Python
people are even remotely interested in having this functionality (only
SUSE seems to agree with Debian). So the chances of Python coming up
with a directory they like for this are very slim.
The best solution that no one disagreed with was placing /usr/local
after everything else on sys.path so that you can't overwrite stuff
installed to /usr without using PYTHONPATH (and knowing what you're
doing).
This could be achieved easily with this patch to
/usr/lib/python2.5/site.py:
--- site.py 2008-04-23 20:51:34.0 +0100
+++ site.py 2008-04-23 20:51:42.0 +0100
@@ -171,7 +171,7 @@
prefixes = [sys.prefix]
if sys.exec_prefix != sys.prefix:
prefixes.append(sys.exec_prefix)
-prefixes.insert(0, '/usr/local')
+prefixes.append('/usr/local')
for prefix in prefixes:
if prefix:
if sys.platform in ('os2emx', 'riscos'):
Would this be a suitable solution for Debian?
Regards
Floris
[1] Seeming to conclude around here:
http://mail.python.org/pipermail/distutils-sig/2008-April/009391.html
--
Debian GNU/Linux -- The Power of Freedom
www.debian.org | www.gnu.org | www.kernel.org
--
To UNSUBSCRIBE, email to [EMAIL PROTECTED]
with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: handling /usr/local/lib/python2.x/site-packages in sys.path
Barry Warsaw wrote: This is Debian policy (which is fine), but I don't think all distros agree. I'm not a distro guy though. :) Mattias, didn't the Fedora guys say they were going to try to create a mailing list for discussing these kinds of cross-distro issues? There's http://lists.freedesktop.org/mailman/listinfo/distributions, you may want to raise this issue there. Cheers, Emilio signature.asc Description: OpenPGP digital signature
Re: handling /usr/local/lib/python2.x/site-packages in sys.path
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Mar 11, 2008, at 6:36 PM, Floris Bruynooghe wrote: On Tue, Mar 11, 2008 at 09:45:21AM -0400, Barry Warsaw wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Mar 11, 2008, at 3:43 AM, Matthias Klose wrote: Currently Debian's python has /usr/local/lib/python2.x/site-packages in sys.path allowing for installation of local modules. Barry did point out that this conflicts with a python installation, where /usr/local is the default prefix, and the system python gets modules from the local installation. Some suggestions were made to fix this: - add an env var to not include /usr/local/lib/python2.x/site- packages when the env var is set. Keeps standard behaviour without modifications and allows people to remove it from sys.path. But requires the user to know about that option. - add another path (e.g. /usr/local/python/lib2.x/site-packages), and remove the /usr/local/lib/python2.x/site-packages path after the next release. Does provide an upgrade path, but doesn't solve the probem immediately. Thanks for raising this issue Matthias. There are a couple of other points I'd like to make. This isn't /just/ a problem for Python experts or developers. Sometimes we recommend that Ordinary Users install Python from source, say to use an application that isn't supported by a distro's packaging system. There can be lots of reasons for this: maybe the app is in beta testing, maybe they want to try things out without affecting the system version, etc. Python developers may not be Debian experts and may not know about this system peculiarity. So when we tell people just do configure; make; make install we can actually do inadvertent damage to their Debian system. Problem is that both are very natural, having local packages being picked up by the system python in /usr/local/lib/python2.X/site-packages as well as being able to install a new python with a prefix of /usr/local. I asked quite a few Python people about this at Pycon 2008, and I raised it in a Python packaging session with Matthias and a few Fedora people. I actually think this is natural only for Debian users, and not the wider Python or Linux crowd. The Fedora people don't do this, nor does Gentoo AFAICT. It surprised quite a few Python folks I spoke with, even some that use Debian or Ubuntu. Given this choice I'd prefer the second option as well, it seems more natural then the first and an upgrade path is important (Wichert's argument about the first having problems with su is also important). It would be good if the distutils could be changed to install local packages there by default then, so as to provide the least surprises: just python setup.py install as root would do the correct thing. I agree that for Debian and Ubuntu, where this tradition is firmly ingrained, the second option makes the most sense. It lets you have the functionality you want without breaking the expectations of the wider Python community. Everyone can live together peacefully! :) I'm not sure your other suggestion make sense from a default distutils perspective. The reason is that distutils doesn't see the world that way, and it really doesn't know anything about the rest of the file system, other than the paths it gets from Python or that you tell it about. So having it install in Python's own site-packages by default still makes sense, but it should be easily configurable to do otherwise, either through command line switch or other mechanism. That's the case today though :). When looking at the grand scale of things the change for the system Python to look for local packages in /usr/local/python/lib2.X/site-packages (or /usr/local/python/site-packages2.X?[1]) is probably best done upstream so it is on all systems the same. The build system could do this when a prefix of /usr is passed to configure. I don't think it makes sense to get this into upstream because this is really a distro question. What about MacPorts installing into /opt/ local? What about if I install from source into ~barry/development/ python2.6? None of those should have anything to do with /usr/local. What confuses me is that this doesn't appear to be a convenience based on permissions, It has nothing to do with permissions. The sysadmin should never install things into /usr/ directly, /usr/local/ is their playground. This is Debian policy (which is fine), but I don't think all distros agree. I'm not a distro guy though. :) Mattias, didn't the Fedora guys say they were going to try to create a mailing list for discussing these kinds of cross-distro issues? I'm also uncomfortable with the way that Debian accomplishes this: it hacks Python's standard site.py. There really should be a better way to do this (I have to think about this a bit though). In the python we ship to our customers (living in /opt/vendor/python2X as per FHS) we
Re: handling /usr/local/lib/python2.x/site-packages in sys.path
Barry Warsaw [EMAIL PROTECTED] writes: On Mar 11, 2008, at 6:36 PM, Floris Bruynooghe wrote: The sysadmin should never install things into /usr/ directly, /usr/local/ is their playground. This is Debian policy (which is fine), but I don't think all distros agree. Those distros that claim to adhere to the FHS agree. /usr is shareable, read-only data. That means that /usr should be shareable between various FHS-compliant hosts and must not be written to. Any information that is host-specific or varies with time is stored elsewhere. URL:http://www.pathname.com/fhs/pub/fhs-2.3.html#THEUSRHIERARCHY The /usr/local hierarchy is for use by the system administrator when installing software locally. It needs to be safe from being overwritten when the system software is updated. It may be used for programs and data that are shareable amongst a group of hosts, but not found in /usr. URL:http://www.pathname.com/fhs/pub/fhs-2.3.html#USRLOCALLOCALHIERARCHY -- \ “The cost of education is trivial compared to the cost of | `\ ignorance.” —Thomas Jefferson | _o__) | Ben Finney -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: handling /usr/local/lib/python2.x/site-packages in sys.path
Barry Warsaw wrote: It also makes Debian the odd man out. Instructions we publish for every other *nix have to have a caveat or FAQ for Debian's (and derivatives) difference. These can go unnoticed until things break, and then we can get difficult to debug problem reports. An experienced helper will be conditioned to first ask Are you on Debian or Ubuntu? Well, you have to do things differently there. This is a point of perspective. From my point of view python is the odd one out: every other tool and compiler seems to look in both /usr/local and /usr *except* for (non-Debian) python. That would be highly confusing for me, and I fail to understand why python made that choice. Debian has always setup /usr/local so that anyone in the staff group can write to it. This is very convenient: it gives you a clear point of demarkation between OS-managed applications, which install in /usr, and manually managed applications which install in /usr/local and can be installed by trusted non-root users. I don't know why your Ubuntu system only makes /usr/local writable for root, to me that sounds like a bug in Ubuntu. I find virtualenv to be a better tool to setup local python environments: it does not require people to recompile all of python while still providing a clean environment to work in. And as far as I can see it is a good solution for almost all use cases where you currently tell people to compile a local python version. Wichert. -- Wichert Akkerman [EMAIL PROTECTED] It is simple to make things. http://www.wiggy.net/ It is hard to make things simple. -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: handling /usr/local/lib/python2.x/site-packages in sys.path
Matthias Klose wrote: Currently Debian's python has /usr/local/lib/python2.x/site-packages in sys.path allowing for installation of local modules. Barry did point out that this conflicts with a python installation, where /usr/local is the default prefix, and the system python gets modules from the local installation. In other words you are trying to make things a bit easier for people who are a) expert enough to build and use their own python version and b) notice things breaking because they use a site-packages path which is also used by the system python. I can sympathise with building your own python (Debian's python uses twice the amount of memory needed for most unicode-using applications due to the use of UCS4 for example) but I think people who know enough to do that also know enough to deal with the resulting possible conflict over /usr/local/lib/python2.x use. Personally I have a number of local python's but they are all isolated in zc.buildout instances. Some suggestions were made to fix this: - add an env var to not include /usr/local/lib/python2.x/site-packages when the env var is set. Keeps standard behaviour without modifications and allows people to remove it from sys.path. But requires the user to know about that option. That makes things unpredicatable: use su or something else that can affect the environment and suddenly python behaves quite differently. That does not feel like a good idea. - add another path (e.g. /usr/local/python/lib2.x/site-packages), and remove the /usr/local/lib/python2.x/site-packages path after the next release. Does provide an upgrade path, but doesn't solve the probem immediately. That just means you will break people's python at a later point again. That certainly can not be the right thing to do. Iff we really must come up with a way to make it a bit easier for people to install their own python version make it a system-wide switch in /etc/default/python2.x and keep the default behaviour as it is now so you will not break existing installations. Wichert. -- Wichert Akkerman [EMAIL PROTECTED]It is simple to make things. http://www.wiggy.net/ It is hard to make things simple. -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: handling /usr/local/lib/python2.x/site-packages in sys.path
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Mar 11, 2008, at 3:58 AM, Wichert Akkerman wrote: - add another path (e.g. /usr/local/python/lib2.x/site-packages), and remove the /usr/local/lib/python2.x/site-packages path after the next release. Does provide an upgrade path, but doesn't solve the probem immediately. That just means you will break people's python at a later point again. That certainly can not be the right thing to do. Depends on how you look at it I guess. I see it as eventually fixing everyone's Python ;). Iff we really must come up with a way to make it a bit easier for people to install their own python version make it a system-wide switch in /etc/default/python2.x and keep the default behaviour as it is now so you will not break existing installations. I'm not sure what the it is you want to make a system-wide switch. Would that be which Python to use, or whether to add the sys.path hack? I don't know who's going to be at Pycon later this week, but if enough distro people are around, I'd be very happy to have a discussion about how to make this more consistent across distros, more amenable to users and developers, and how to make things less mysterious, confusing, or fragile. Given how many system tools are being built in Python now (a good thing!), there should be some agreement on the right way to do it. I'll be at Pycon and would be willing to chat about this at a BOF, over lunch, or over beers. :) - -Barry -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.8 (Darwin) iQCVAwUBR9aPNHEjvBPtnXfVAQJCzgP/QaadzyGxnvA6WqT8B3QsouSVvZvsmg9n LAOnoLYzZOhcLzSUlYiukPcU6DMX8HAA8AaAvwpiyCLmk6IZDGMJxDZH+R8u4+w7 9E4KRdt3yxa5nbcJAMa/fWzjpDTlXSecZBAHw7IAYngy55+fv7cJ7UvWgNqsPdIB h4mUoEU32CU= =I8kv -END PGP SIGNATURE- -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: handling /usr/local/lib/python2.x/site-packages in sys.path
On Tue, Mar 11, 2008 at 08:43:06AM +0100, Matthias Klose wrote: Currently Debian's python has /usr/local/lib/python2.x/site-packages in sys.path allowing for installation of local modules. Barry did point out that this conflicts with a python installation, where /usr/local is the default prefix, and the system python gets modules from the local installation. Some suggestions were made to fix this: Well, I would expect that package I install in /usr/local are found automatically by the standard Python install. I think locally installing packages is a far more common use case than installing a hand-compiled python. It would be very unusual and inconvenient if installing modules with python setup.py --prefix=/usr/local wouldn't work with the standard Python installation. - Sebastian -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: handling /usr/local/lib/python2.x/site-packages in sys.path
[Matthias Klose, 2008-03-11] - add another path (e.g. /usr/local/python/lib2.x/site-packages), and remove the /usr/local/lib/python2.x/site-packages path after the next release. Does provide an upgrade path, but doesn't solve the probem immediately. I would choose this option or even remove /usr/local/* from sys.path completely. If one wants to play with unsupported Python modules or Eggs, he should sys.path.append(his_path) by hand, IMHO. If we additionally force ez_* mess to install to /usr/local/ by default, we will get rid of one of the most common problems I receive from users of my packages. -- -=[ Piotr Ozarowski ]=- -=[ http://www.ozarowski.pl ]=- pgp5fH41nar8Z.pgp Description: PGP signature
Re: handling /usr/local/lib/python2.x/site-packages in sys.path
On Tue, Mar 11, 2008 at 06:58:56PM +0100, Piotr Ożarowski wrote: I would choose this option or even remove /usr/local/* from sys.path completely. If one wants to play with unsupported Python modules or Eggs, he should sys.path.append(his_path) by hand, IMHO. I would not consider installing packages that are not packaged in Debian playing. Installing such packages is a common use case for system administrators and advanced users. With the same reasoning we could remove /usr/local/lib from /etc/ld.so.conf. - Sebastian -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: handling /usr/local/lib/python2.x/site-packages in sys.path
Matthias Klose wrote: - add an env var to not include /usr/local/lib/python2.x/site-packages when the env var is set. Keeps standard behaviour without modifications and allows people to remove it from sys.path. But requires the user to know about that option. I would much prefer this. Debian users with local (=unpackaged) packages are probably much more common than Debian users with (=unpackaged) versions of all python, and I would rather reasonably support those than leaving them in the cold. Quite frankly, installing stuff that is also present in the system under user local is a recipe for disaster (also happens with libraries) and rather hard to cater for. - add another path (e.g. /usr/local/python/lib2.x/site-packages), and remove the /usr/local/lib/python2.x/site-packages path after the next release. Does provide an upgrade path, but doesn't solve the probem immediately. No. Let's not break stuff for people that use the software in Debian that they got from Debian. Quite frankly, the use case seems a bit of bad practice. Installing to /usr/local software to compete with that in /usr leads to all sorts of breakage where one doesn't expect it and if python.org needs people to install experimental versions they should either recommend chroots / test machines or provide packages. Anyone capable of installing a local version of python is also able to run debootstrap to create a chroot. Kind regards T. (who used to maintain a set of libraries where local installations caused no end of trouble for users, maintainers, and upstreams) -- Thomas Viehmann, http://thomas.viehmann.net/ -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: handling /usr/local/lib/python2.x/site-packages in sys.path
[Sebastian Rittau, 2008-03-11] On Tue, Mar 11, 2008 at 06:58:56PM +0100, Piotr Ożarowski wrote: I would choose this option or even remove /usr/local/* from sys.path completely. If one wants to play with unsupported Python modules or Eggs, he should sys.path.append(his_path) by hand, IMHO. I would not consider installing packages that are not packaged in Debian playing. Installing such packages is a common use case for system administrators and advanced users. With the same reasoning we could remove /usr/local/lib from /etc/ld.so.conf. true. /me still doesn't know how to (at the same time) eat a cookie (get rid of Egg problems) and still be able to keep it (give advanced users/administrators possibility to add local modifications). -- -=[ Piotr Ozarowski ]=- -=[ http://www.ozarowski.pl ]=- pgptrogCSkHrZ.pgp Description: PGP signature
Re: handling /usr/local/lib/python2.x/site-packages in sys.path
On Tue, Mar 11, 2008 at 09:45:21AM -0400, Barry Warsaw wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Mar 11, 2008, at 3:43 AM, Matthias Klose wrote: Currently Debian's python has /usr/local/lib/python2.x/site-packages in sys.path allowing for installation of local modules. Barry did point out that this conflicts with a python installation, where /usr/local is the default prefix, and the system python gets modules from the local installation. Some suggestions were made to fix this: - add an env var to not include /usr/local/lib/python2.x/site-packages when the env var is set. Keeps standard behaviour without modifications and allows people to remove it from sys.path. But requires the user to know about that option. - add another path (e.g. /usr/local/python/lib2.x/site-packages), and remove the /usr/local/lib/python2.x/site-packages path after the next release. Does provide an upgrade path, but doesn't solve the probem immediately. Thanks for raising this issue Matthias. There are a couple of other points I'd like to make. This isn't /just/ a problem for Python experts or developers. Sometimes we recommend that Ordinary Users install Python from source, say to use an application that isn't supported by a distro's packaging system. There can be lots of reasons for this: maybe the app is in beta testing, maybe they want to try things out without affecting the system version, etc. Python developers may not be Debian experts and may not know about this system peculiarity. So when we tell people just do configure; make; make install we can actually do inadvertent damage to their Debian system. Problem is that both are very natural, having local packages being picked up by the system python in /usr/local/lib/python2.X/site-packages as well as being able to install a new python with a prefix of /usr/local. Given this choice I'd prefer the second option as well, it seems more natural then the first and an upgrade path is important (Wichert's argument about the first having problems with su is also important). It would be good if the distutils could be changed to install local packages there by default then, so as to provide the least surprises: just python setup.py install as root would do the correct thing. When looking at the grand scale of things the change for the system Python to look for local packages in /usr/local/python/lib2.X/site-packages (or /usr/local/python/site-packages2.X?[1]) is probably best done upstream so it is on all systems the same. The build system could do this when a prefix of /usr is passed to configure. This can not be done by using a system wide configuration file IIRC, since the Python make system will pick up those configuration files hence breaking the /usr/local install again as well as generating FTBFS on systems that have python already installed during the build[0]. This is possibly a bug in the Python make system though, what is the rationale for picking up the distutils configuration files for building python itself? Surely all this info is passed to configure? What confuses me is that this doesn't appear to be a convenience based on permissions, It has nothing to do with permissions. The sysadmin should never install things into /usr/ directly, /usr/local/ is their playground. I'm also uncomfortable with the way that Debian accomplishes this: it hacks Python's standard site.py. There really should be a better way to do this (I have to think about this a bit though). In the python we ship to our customers (living in /opt/vendor/python2X as per FHS) we place a .pth file in the site-packages directory to accomplish extra locations on sys.path (pointing somewhere else in our /opt/vendor hierarchy). Maybe /usr/lib/python2.X/site-packages/debian.pth would be a better way to handle this on Debian? Or are there reasons not to do this? Although the suggestion I make above, changing it upstream, would make this unnecessary. [As an aside, I find what happens with python-support much worse. I regularly waste time re-discovering that a whole load of system installed packages can't be found in /usr/lib/python2.X/site-packages/ but live somewhere in /var/lib/python-support/. It's a violation of the FHS as well, /var/lib/ is for program state information, .pyc's are not state).] Regards Floris [0] This has bitten me in the past with FTBFS when having a ~/.pydistutils.cfg defining a `home=~' and trying to rebuild the Debian Python package. I think it also happens when trying to build a new upstream python with an existing distutils configuration somewhere. [1] Both violate FHS though. The directories allowed in /usr/local after a fresh install are limited (http://www.pathname.com/fhs/pub/fhs-2.3.html#USRLOCALLOCALHIERARCHY). But /usr/local/lib/python/site-packages2.X does seem a little cumbersome and confusing (since a local python will use
