I'm trying to add a lightweight "virtualenv" to a local build system
using PYTHONUSERBASE. Unfortunately the package maintainers for
distutils and setuptools broke it on my Ubuntu laptop. I have
incontrovertable proof!
PYTHONUSERBASE doesn't scan the exact directory you set it to; you're
pointing it at a "prefix" directory, and it looks in directories under
there. Well, *directory*, really: PYTHONUSERBASE only causes CPython to
scan one additional directory for packages. That directory is:
"{prefix}/lib/python2.6/site-packages". This is true whether you use
the Debian packaged build or if you build Python yourself from source.
AFAICT distutils uses distutils.sysconfig.get_python_lib() to decide
where you should install a site package. Depending on the inputs, it
will give you one of two directories. If it's a "standard library",
it'll give you "{prefix}/lib/python2.6".
However! If you call get_python_lib() saying it's *not* a standard
library, you get different results. If you build CPython from scratch
you'll get "{prefix}/lib/python2.6/site-packages". If you use the
Ubuntu Python 2.6 package, you'll get
"{prefix}/lib/python2.6/dist-packages". But PYTHONUSERBASE still only
looks in the "site-packages" directory. It ignores this directory and
therefore doesn't pick up your packages.
A more obscure but more widespread bug: consider that PYTHONUSERBASE
also ignores the "standard library" directory returned by
get_python_lib(). If you wanted to have your own local version of a
"standard library", you couldn't put it in "{prefix}/lib/python2.6".
This is true whether you build CPython from scratch or use the Ubuntu
packages.
setuptools under Ubuntu breaks PYTHONUSERBASE in a similar but different
way. If you install the setuptools egg by hand, Setuptools will install
packages into "{prefix}/lib/python2.6/site-packages". If you use the
Ubuntu package for setuptools, setuptools will install packages into
"{prefix}/local/lib/python2.6/dist-packages". Again, this is a
directory PYTHONUSERBASE ignores.
In my opinion the best way to fix this would be to add a new environment
variable and deprecate the old one. I nominate "PYTHONUSERBASES"--note
the plural. PYTHONUSERBASES would support a
local-path-separator-separated list of directories, all of which would
be used as site package directories. For each directory on the path, we
would add *the same list of subdirectories* that site.addsitepackages()
does.
Failing that, PYTHONUSERBASE should be at least be changed so it adds
the same directories as site.addsitepackages().
I would be happy to contribute a patch to do either of these, for Python
2.x and 3.x.
What follows is my test case where I figured all this out. I started by
creating a PYTHONUSERBASE directory, which in my late-night hacking
fever I called "/home/larry/pwned". Inside I created subdirectories
matching every directory I'd ever seen scanned for site packages, like
"lib/python2.6/site-packages". Then for each directory I added a
do-nothing module named for the directory it was in, with a "UB" on the
front so I knew it came from my PYTHONUSERBASE. For example, one file
was called
"/home/larry/pwned/local/lib/python2.6/dist-packages/UBlocallibpythondistpackages.py".
Next, I added those same files to the equivalent directories in "/usr",
only with "SYSTEM" on the front. For instance,
"/usr/lib/python2.6/site-packages/SYSTEMlibpythonsitepackages.py"
Next, I built Python 2.6 from source, with a prefix directory of
"/home/larry/src/python/userbase/release", and installed setuptools from
the egg on cheeseshop. I then added these same files one last time, and
again with SYSTEM on the front, to my locally-built Python's prefix
directory.
Finally I wrote a Python script that tried to import every one of these
modules, and ran it as follows:
% PYTHONUSERBASE=/home/larry/pwned python findcookies.py
% PYTHONUSERBASE=/home/larry/pwned ./python findcookies.py
Here's the output. First, the results from running my built-from-source
Python. Hopefully you're reading this in a fixed-point font on a wide
screen:
--
% PYTHONUSERBASE=/home/larry/pwned
/home/larry/src/python/userbase/release/bin/python
/home/larry/findcookies.py
------------------------------------------------------------------------------
Where does distutils say we should install?
Calling distutils.sysconfig.get_python_lib()
with two different prefixes (sys.prefix and "/home/larry/pwned")
and all combinations of its two boolean arguments:
du.sc.gpl(True , True , '/home/larry/src/python/userbase/release') =
'/home/larry/src/python/userbase/release/lib/python2.6'
du.sc.gpl(True , True , '/home/larry/pwned') =
'/home/larry/pwned/lib/python2.6'
du.sc.gpl(True , False, '/home/larry/src/python/userbase/release') =
'/home/larry/src/python/userbase/release/lib/python2.6/site-packages'
du.sc.gpl(True , False, '/home/larry/pwned') =
'/home/larry/pwned/lib/python2.6/site-packages'
du.sc.gpl(False, True , '/home/larry/src/python/userbase/release') =
'/home/larry/src/python/userbase/release/lib/python2.6'
du.sc.gpl(False, True , '/home/larry/pwned') =
'/home/larry/pwned/lib/python2.6'
du.sc.gpl(False, False, '/home/larry/src/python/userbase/release') =
'/home/larry/src/python/userbase/release/lib/python2.6/site-packages'
du.sc.gpl(False, False, '/home/larry/pwned') =
'/home/larry/pwned/lib/python2.6/site-packages'
------------------------------------------------------------------------------
Where does setup_tools say we should install?
Expanding
setuptools.command.easy_install.easy_install.INSTALL_SCHEMES[os.name]["install_dir"]
with two different prefixes (sys.prefix and "/home/larry/pwned"):
/home/larry/src/python/userbase/release/lib/python2.6/site-packages
/home/larry/pwned/lib/python2.6/site-packages
------------------------------------------------------------------------------
Finally: trying to load a module from every possible site-packages
directory.
There are five columns in the output; here's what they mean.
+----------- Marked with "*" if distutils.sysconfig.get_python_lib
| told us to use this directory.
|
| +--------- Marked with "EZ" if
setuptools.command.easy_install.easy_install.INSTALL_SCHEMES
| | told us to use this directory.
| |
| | +------ Could we load this module?
| | |
| | | +--- What silly name did I give +--- What directory did I
| | | | to this module? | stick this module
into?
| | | | |
v v v v v
* True SYSTEMlibpython
/home/larry/src/python/userbase/release/lib/python2.6
True SYSTEMlibsitepython
/home/larry/src/python/userbase/release/lib/site-python
False SYSTEMlibdistpython
/home/larry/src/python/userbase/release/lib/dist-python
* EZ True SYSTEMlibpythonsitepackages
/home/larry/src/python/userbase/release/lib/python2.6/site-packages
False SYSTEMlibpythondistpackages
/home/larry/src/python/userbase/release/lib/python2.6/dist-packages
False SYSTEMlocallibpythondistpackages
/home/larry/src/python/userbase/release/local/lib/python2.6/dist-packages
* False UBlibpython
/home/larry/pwned/lib/python2.6
False UBlibsitepython
/home/larry/pwned/lib/site-python
False UBlibdistpython
/home/larry/pwned/lib/dist-python
* EZ True UBlibpythonsitepackages
/home/larry/pwned/lib/python2.6/site-packages
False UBlibpythondistpackages
/home/larry/pwned/lib/python2.6/dist-packages
False UBlocallibpythondistpackages
/home/larry/pwned/local/lib/python2.6/dist-packages
--
Next, the output from the CPython and setuptools from the Ubuntu packages:
--
% PYTHONUSERBASE=/home/larry/pwned /usr/bin/python
/home/larry/findcookies.py
------------------------------------------------------------------------------
Where does distutils say we should install?
Calling distutils.sysconfig.get_python_lib()
with two different prefixes (sys.prefix and "/home/larry/pwned")
and all combinations of its two boolean arguments:
du.sc.gpl(True , True , '/usr') =
'/usr/lib/python2.6'
du.sc.gpl(True , True , '/home/larry/pwned') =
'/home/larry/pwned/lib/python2.6'
du.sc.gpl(True , False, '/usr') =
'/usr/lib/python2.6/dist-packages'
du.sc.gpl(True , False, '/home/larry/pwned') =
'/home/larry/pwned/lib/python2.6/dist-packages'
du.sc.gpl(False, True , '/usr') =
'/usr/lib/python2.6'
du.sc.gpl(False, True , '/home/larry/pwned') =
'/home/larry/pwned/lib/python2.6'
du.sc.gpl(False, False, '/usr') =
'/usr/lib/python2.6/dist-packages'
du.sc.gpl(False, False, '/home/larry/pwned') =
'/home/larry/pwned/lib/python2.6/dist-packages'
------------------------------------------------------------------------------
Where does setup_tools say we should install?
Expanding
setuptools.command.easy_install.easy_install.INSTALL_SCHEMES[os.name]["install_dir"]
with two different prefixes (sys.prefix and "/home/larry/pwned"):
/usr/local/lib/python2.6/dist-packages
/home/larry/pwned/local/lib/python2.6/dist-packages
------------------------------------------------------------------------------
Finally: trying to load a module from every possible site-packages
directory.
There are five columns in the output; here's what they mean.
+----------- Marked with "*" if distutils.sysconfig.get_python_lib
| told us to use this directory.
|
| +--------- Marked with "EZ" if
setuptools.command.easy_install.easy_install.INSTALL_SCHEMES
| | told us to use this directory.
| |
| | +------ Could we load this module?
| | |
| | | +--- What silly name did I give +--- What directory did I
| | | | to this module? | stick this module
into?
| | | | |
v v v v v
* True SYSTEMlibpython /usr/lib/python2.6
False SYSTEMlibsitepython /usr/lib/site-python
True SYSTEMlibdistpython /usr/lib/dist-python
False SYSTEMlibpythonsitepackages
/usr/lib/python2.6/site-packages
* True SYSTEMlibpythondistpackages
/usr/lib/python2.6/dist-packages
EZ True SYSTEMlocallibpythondistpackages
/usr/local/lib/python2.6/dist-packages
* False UBlibpython
/home/larry/pwned/lib/python2.6
False UBlibsitepython
/home/larry/pwned/lib/site-python
False UBlibdistpython
/home/larry/pwned/lib/dist-python
True UBlibpythonsitepackages
/home/larry/pwned/lib/python2.6/site-packages
* False UBlibpythondistpackages
/home/larry/pwned/lib/python2.6/dist-packages
EZ False UBlocallibpythondistpackages
/home/larry/pwned/local/lib/python2.6/dist-packages
--
For what it's worth, here's my horrible hacked-together script:
--
import distutils.sysconfig
import os
import sys
paths = {}
print """
%% PYTHONUSERBASE=%s %s %s
------------------------------------------------------------------------------
Where does distutils say we should install?
Calling distutils.sysconfig.get_python_lib()
with two different prefixes (sys.prefix and "/home/larry/pwned")
and all combinations of its two boolean arguments:
""".strip() % (os.environ["PYTHONUSERBASE"], sys.executable, sys.argv[0])
print
for platformSpecific in (True, False):
for standardLib in (True, False):
for prefix in (sys.prefix, "/home/larry/pwned"):
path = distutils.sysconfig.get_python_lib(platformSpecific,
standardLib, prefix)
print (" du.sc.gpl(" + str(platformSpecific).ljust(5) + ",
" + str(standardLib).ljust(5) + ", " + repr(prefix) + ")").ljust(69),
"=", repr(path)
paths[path] = "* "
print
print
print """
------------------------------------------------------------------------------
Where does setup_tools say we should install?
Expanding
setuptools.command.easy_install.easy_install.INSTALL_SCHEMES[os.name]["install_dir"]
with two different prefixes (sys.prefix and "/home/larry/pwned"):
""".strip()
print
import setuptools.command.easy_install as ei
for where, base in (("system", sys.prefix), ("userbase",
"/home/larry/pwned")):
path =
ei.easy_install.INSTALL_SCHEMES[os.name]["install_dir"].replace("$base",
base).replace("$py_version_short", "2.6")
print " ", path
if path in paths:
paths[path] = "* EZ"
else:
paths[path] = " EZ"
print
print """
------------------------------------------------------------------------------
Finally: trying to load a module from every possible site-packages
directory.
There are five columns in the output; here's what they mean.
+----------- Marked with "*" if distutils.sysconfig.get_python_lib
| told us to use this directory.
|
| +--------- Marked with "EZ" if
setuptools.command.easy_install.easy_install.INSTALL_SCHEMES
| | told us to use this directory.
| |
| | +------ Could we load this module?
| | |
| | | +--- What silly name did I give +--- What directory did I
| | | | to this module? | stick this module
into?
| | | | |
v v v v v
""".strip()
for prefix, pathprefix, where in (("SYSTEM",
os.path.normpath(sys.prefix) + "/", "system "), ("UB",
"/home/larry/pwned/", "userbase")):
for _name, path in (
("libpython", "lib/python2.6"),
("libsitepython", "lib/site-python"),
("libdistpython", "lib/dist-python"),
("libpythonsitepackages", "lib/python2.6/site-packages"),
("libpythondistpackages", "lib/python2.6/dist-packages"),
# nobody ever told me to use one of these four, so I'm removing 'em.
# ("locallibpython", "local/lib/python2.6"),
# ("locallibsitepython", "local/lib/site-python"),
# ("locallibdistpython", "local/lib/dist-python"),
# ("locallibpythonsitepackages",
"local/lib/python2.6/site-packages"),
("locallibpythondistpackages", "local/lib/python2.6/dist-packages"),
):
name = prefix + _name
fullpath = os.path.join(pathprefix, path)
filename = os.path.join(fullpath, name + ".py")
assert os.path.isfile(filename), "Your test is wrong, %s does
not exist!" % repr(filename)
s = "import {0}".format(name)
try:
exec s
worked = True
except ImportError:
worked = False
starred = paths.get(fullpath, " ")
print " ", starred, str(worked).ljust(6), name.ljust(35), fullpath
if 0:
#print path
if 1:
name = name.replace("UB", "SYSTEM")
if not os.path.isdir(path):
os.makedirs(path)
filename = path + "/" + name + ".py"
filename = filename.replace("/python/", "/python2.6/")
#print filename
#continue
f = open(filename, "wt")
f.write('cookie = "PIXIE UNICORN"\n')
f.close()
print
--
If you read this far, I congratulate you on your powers of concentration.
Hope this helps,
/larry/
_______________________________________________
Distutils-SIG maillist - [email protected]
http://mail.python.org/mailman/listinfo/distutils-sig
