[issue5819] Add PYTHONPREFIXES environment variable
Andrew Svetlov added the comment: Close issue as superseded by venv package. -- resolution: - out of date stage: - committed/rejected status: open - closed ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue5819 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue5819] Add PYTHONPREFIXES environment variable
Christopher Dunn cdunn2...@gmail.com added the comment: I agree. venv solves this problem and more. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue5819 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue5819] Add PYTHONPREFIXES environment variable
Andrew Svetlov andrew.svet...@gmail.com added the comment: As PEP 405 has been implemented in Python 3.3 this issue can be closed I think. -- nosy: +asvetlov ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue5819 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue5819] Add PYTHONPREFIXES environment variable
Éric Araujo mer...@netwok.org added the comment: See also #1298835 and PEP 405. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue5819 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue5819] Add PYTHONPREFIXES environment variable
Changes by Éric Araujo mer...@netwok.org: -- nosy: +barry, eric.araujo versions: +Python 3.2 -Python 3.1 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue5819 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue5819] Add PYTHONPREFIXES environment variable
Barry A. Warsaw ba...@python.org added the comment: I'll look at this in more detail, but I can at least answer one question right now: The bona-fide known bug: the Debian package mantainer for Python decided to change site-packages to dist-packages in 2.6, for reasons I still don't quite understand. This was done as an accommodation to upstream Python developers. A from-source build of Python installs into /usr/local by default, with a site-packages situated there. But the Debian interpretation of the FHS states that site-installed (i.e. not through the package manager) add-ons to Python should also live in /usr/local, and /usr/local/lib/pythonX.Y/site-packages was chosen as the most obvious place for that. Unfortunately, this meant that if you installed Python from source using the defaults, it was possible to clobber either your system Python or your from-source installed Python by installing third party packages to an unintentionally shared directory. This was a real problem. Now, the Debian answer (probably rightly so) was that Python's from-source defaults should install into /opt not /usr/local, but this broke a decade's (at least) convention from upstream. The compromise was to keep Debian's /usr/local interpretation of the FHS, but to choose a directory that would not conflict with a from-source installation of Python. Thus dist-packages was chosen. To keep the system Python consistent, Python packages installed via apt are installed to /usr/lib/pythonX.Y/dist-packages too. I've had discussions with developers on both sides. It's not an ideal solution to anyone, but I thought it was the best compromise available at that time, and still do. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue5819 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue5819] Add PYTHONPREFIXES environment variable
Christopher Dunn cdunn2...@gmail.com added the comment: I am not sure that this guy's idea is good, but I think that he deserves more attention. In comments elsewhere on the web, I noticed that people thought the PYTHONUSERBASE site-packages directory could contain .pth files which would serve the same purpose as his proposed environment variable. However, .pth files are not recursive; a .pth file in one directory does not cause .pth files to be processed in the directories named by its contents. I agree with ianb that this is a poor way to mimic virtualenv. There is a difference between user additions -- which should affect *all* python code that he uses -- and separate Python installations. There might be other reasons for multiple user site-packages directories. For example, Python lacks Perl's architecture awareness. Perl can load from lib/ lib/5.10.0/ lib/5.10.0/linux-x86 lib/5.10.0/linux-x86/auto and a few other combinations. Python also lacks a PERL5OPT equivalent. I am forced to use the PYTHONUSERBASE mechanism to simulate that behavior (which I use to turn on code coverage everywhere during testing) but I only get to set that once. -- nosy: +cdunn2001 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue5819 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue5819] Add PYTHONPREFIXES environment variable
New submission from Larry Hastings la...@hastings.org:
The attached patch adds support for a new environment variable,
PYTHONPREFIXES. PYTHONPREFIXES is similar to PYTHONUSERBASE: it lets
you add prefix directories to be culled for site packages. It differs
from PYTHONUSERBASE in three ways:
* PYTHONPREFIXES has an empty default value. PYTHONUSERBASE has a
default, e.g. ~/.local on UNIX-like systems.
* PYTHONPREFIXES supports multiple directories, separated by the
site-specific directory separator character (os.pathsep). Earlier
directories take precedence. PYTHONUSERBASE supports specifying
at most one directory.
* PYTHONPREFIXES adds its directories to site.PREFIXES, so it reuses
the existing mechanisms for site package directories, exactly
simulating a real prefix directory. PYTHONUSERBASE only adds a
single directory, using its own custom code path.
This last point bears further discussion. PYTHONUSERBASE's custom code
to inspect only a single directory has resulted in at least one bug, if
not more, as follows:
* The bona-fide known bug: the Debian package mantainer for Python
decided to change site-packages to dist-packages in 2.6,
for reasons I still don't quite understand. He made this change in
site.addsitepackages and distutils.sysconfig.get_python_lib, and
similarly in setuptools, but he missed changing it in
site.addusersitepackages. This meant that if you used setup.py to
install a package to a private prefix directory, PYTHONUSERBASE had
no hope of ever finding the package. (Happily this bug is fixed.)
* I suspect there's a similar bug with PYTHONUSERBASE on the os2emx
and riscos platforms. site.addsitepackages on those platforms
looks in {prefix}/Lib/site-packages, but
site.addusersitepackages looks in
{prefix}/lib/python{version}/site-packages as it does
on any non-Windows platform. Presumably setup.py on those two
platforms installs site packages to the directory site.addsitepackages
inspects, which means that PYTHONUSERBASE doesn't work on those
two platforms.
PYTHONUSERBASE's custom code path to add site package directories is a
source of unnecessary complexity and bugs. I cannot fathom why its
implementors chose this approach; in any case I think reusing
site.addsitepackages is a clear win. I suspect it's too late to change
the semantics of PYTHONUSERBASE to simply call site.addsitepackages,
though if that idea found support I'd be happy to contribute a patch.
A few more notes on PYTHONPREFIXES:
* PYTHONPREFIXES is gated by the exact same mechanisms that shut off
PYTHONUSERBASE.
* Specifying -s on the Python command line shuts it off.
* Setting the environment variable PYTHONNOUSERSITE to a non-empty
string shuts it off.
* If the effective uid / gid doesn't match the actual uid / gid it
automatically shuts off.
* I'm not enormously happy with the name. Until about an hour or two
ago I was calling it PYTHONUSERBASES. I'm open to other
suggestions.
* I'm not sure that PYTHONPREFIX should literally modify site.PREFIXES.
If that's a bad idea I'd be happy to amend the patch so it didn't
touch site.PREFIXES.
* Reaction in python-ideas has been reasonably positive, though I gather
Nick Coughlan and Scott David Daniels think it's unnecessary. (To
read the discussion, search for the old name: PYTHONUSERBASES.)
* Ben Finney prefers a counter-proposal he made in the python-ideas
discussion: change the existing PYTHONUSERBASE to support multiple
directories. I don't like this approach, because:
a) it means you have to explicitly add the local default if you
want to use it, and
b) PYTHONUSERBASE only inspects one directory, whereas PYTHONPREFIX
inspects all the directories Python might use for site packages.
I do admit this approach would be preferable to no change at all.
The attached patch is thrillingly simple and works fine. However it's
not ready to be merged because I haven't touched the documentation. I
figured I'd hold off until I see which way the wind blows.
I'd also be happy to whip out a PEP if that's what is called for.
--
files: lch.pythonprefixes.r71812.diff
keywords: patch
messages: 86352
nosy: lhastings
severity: normal
status: open
title: Add PYTHONPREFIXES environment variable
Added file: http://bugs.python.org/file13745/lch.pythonprefixes.r71812.diff
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue5819
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue5819] Add PYTHONPREFIXES environment variable
Larry Hastings la...@hastings.org added the comment: Whoops, didn't classify the patch before submission. -- components: +Library (Lib) type: - feature request versions: +Python 3.1 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue5819 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue5819] Add PYTHONPREFIXES environment variable
Ian Bicking i...@colorstudy.com added the comment: This has a similar purpose to virtualenv, but using an environmental variable. An earlier package, workingenv, also used an environmental variable, and this led to a set of problems. The biggest problem is that the environmental variable is inherited by subprocesses. This means if you install hg globally, then do subprocess.call(['hg', ...]), then hg will have picked up your local environment. Sometimes this is what you want (e.g., when using ipython) and sometimes not (probably not when using hg). Another problem is that scripts aren't really sticky with respect to the environment. When you install a script using this, it may only work when that same environmental variable is set. But the script remains present and callable regardless. Also, it's hard to mix and match environments in this system. These are real-world problems I encountered with workingenv, and virtualenv has resolved them very reliably by instead using sys.executable to select the environment. This requires some infrastructure in each environment which is unfortunate, but the result is more consistent behavior. -- nosy: +ianb ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue5819 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue5819] Add PYTHONPREFIXES environment variable
Ian Bicking i...@colorstudy.com added the comment: Also with respect to the patch, for consistency there needs to be changes to distutils to make use of this variable. PYTHONUSERBASE included changes so that you can install based on that variable. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue5819 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue5819] Add PYTHONPREFIXES environment variable
Larry Hastings la...@hastings.org added the comment: Thanks for your battle-tested feedback, Mr. Bicking! I reply inline. The biggest problem is that the environmental variable is inherited by subprocesses. [...] Another problem is that scripts aren't really sticky with respect to the environment. [...] These are real-world problems I encountered with workingenv, and virtualenv has resolved them very reliably by instead using sys.executable to select the environment. Excellent points. PYTHONPREFIXES is not a virtualization cure-all; for a complete solution you clearly need an executable file for folks to hang their hat on. However: wouldn't PYTHONPREFIXES greatly simplify virtualenv? All you should need is an executable and a sitecustomize module. Your executable would set PYTHONPREFIXES as makes sense and run Python. Your usercustomize would: * set sys.executable, * set sys.prefix (and maybe sys.exec_prefix), * *unset* PYTHONPREFIXES, and * run the user's real sitecustomize if present. If this works reliably it would obviate most of the work virtualenv currently has to do. In fact I just tried this. A two-line shell script for python3, a twenty-line sitecustomize.py, installed lib/python3.1/config/Makefile and include/python3.1 in my virtualized environment, and I was able to install HeapDict for python3 using its setup.py. (PyPI lies, though; HeapDict doesn't work unmodified in python3. It calls callable(). But once I fixed that it ran fine.) So it passes a smoke-test at least. Also, it's hard to mix and match environments in this system. How so? I thought this proposal made it far easier to mix and match environments. PYTHONPREFIXES is a stack; push and pop environments as you like. Also with respect to the patch, for consistency there needs to be changes to distutils to make use of this variable. PYTHONUSERBASE included changes so that you can install based on that variable. Good call. I found it infuriating that setuptools didn't (still doesn't iirc) understand PYTHONUSERBASE, and would complain that the --prefix directory doesn't support .pth files. If this patch goes further I'll fix up distutils. Thanks again for your feedback! -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue5819 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
