Re: [Distutils] pythonv, take two
Carl Meyer carl at oddbird.net writes: Looks great! I can confirm it works when installed. I'll look into the source-build issues. I've also updated pmv.py at https://gist.github.com/893507 to be closer to what we want for virtualize.py in the stdlib, so that you can do python3.3 -m virtualize /tmp/testenv and have it work: vinay@eta-natty:~/projects/scratch$ python3.3 -m virtualize -h usage: virtualize.py [-h] [--no-distribute] [--no-site-packages] [--clear] ENV_DIR [ENV_DIR ...] positional arguments: ENV_DIR A directory to create the environment in. optional arguments: -h, --help show this help message and exit --no-distribute Don't install Distribute in the virtual environment. --no-site-packages Don't give access to the global site-packages dir to the virtual environment. --clear Delete the environment directory if it already exists. If not specified and the directory exists, an error is raised. vinay@eta-natty:~/projects/scratch$ python3.3 -m virtualize /tmp/testenv Installing distribute .. .. .. .. .done. vinay@eta-natty:~/projects/scratch$ ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] pythonv, take two
Carl Meyer carl at oddbird.net writes: People tend to create a lot of virtualenvs. I don't think copying the entire Python installation is viable (it is slow, and it also would mean that security updates to the base standard library would not be reflected in the virtualenv). You're probably right - I haven't actually tried it. But, as I mentioned, the fact that easy_install itself relies on finding site-packages relative to sys.prefix forces the issue - easy_install (including existing versions of it) needs to just work. Presumably this means that we can't use site_prefix in the short term, and just have it there to allow code to evolve compatibility to it. So everything else just uses site.prefix and site.exec_prefix as it does currently, and we achieve parity with virtualenv as far as compatibility is concerned. Internally, we use sys.installed_prefix and sys.installed_exec_prefix when we need to. We still benefit from the fact that no copying of the standard lib is needed, we're just using the stock site.py etc. BTW you might find this script handy while testing: https://gist.github.com/896831 Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] pythonv, take two
At 02:14 PM 4/1/2011 -0400, Carl Meyer wrote: But, as I mentioned, the fact that easy_install itself relies on finding site-packages relative to sys.prefix forces the issue - easy_install (including existing versions of it) needs to just work. No, it doesn't, actually. easy_install only uses sys.prefix to identify site directories without having to test them. Its install paths actually come from the distutils -- so as long as the distutils install command has the right defaults, easy_install will work just fine. (Assuming sys.executable is correct.) What easy_install does if it's asked to install to a directory that: 1) doesn't look like a site directory (based on sys.prefix), and 2) isn't listed in --site-dirs on the command line or in the config files, is that it runs a test (using sys.executable) to see if .pth files are actually supported in that directory. So, if sys.prefix leads it to believe that the distutils-supplied target directory isn't really a site directory, it checks for itself whether it functions as a site directory. In other words, easy_install will run a bit slower in such a scenario (due to the extra time taken to test), but still function correctly. To stop it taking the extra time for the test, an additional command line option or configuration file option can be given. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] pythonv, take two
On 03/31/2011 04:12 AM, Vinay Sajip wrote:
Carl Meyer carl at oddbird.net writes:
The logical conclusion of that approach, if you want
full compatibility with third-party code, is to just copy the entire
Python installation to the new location.
We do want that compatibility, don't we? Using a virtual environment should
be a
no-brainer, as virtualenv currently is (most of the time). Of course, the
downside of the copying would be around 50MB per virtualenv on Windows, and
much
less where symlinking or hardlinking is available. Not ideal, of course, but
disk space is cheap ;-)
People tend to create a lot of virtualenvs. I don't think copying the
entire Python installation is viable (it is slow, and it also would mean
that security updates to the base standard library would not be
reflected in the virtualenv). If copying the entire base Python install
were an adequate solution, this entire virtual-python proposal would be
moot - you can do that right now and it works fine.
1) Add sys.virtual_prefix (or perhaps sys.site_prefix would be a better
name), and require third-party code to use that (or sysconfig) to find
site-packages, if it wants to be virtual-compatible.
But the success of virtualenv is because you can install just about any
package
in an environment, and packages don't have to do anything to be virtual
compatible.
Sure. virtual-compatible was a poor choice of terms - in the long run,
there should be no such concept. In the short run, we would be
clarifying the semantics of sys.prefix and introducing a new
sys.site_prefix. And in the short run, the effect of code that used
sys.prefix contrary to those semantics would be failure in a virtualenv.
But, as I mentioned, the fact that easy_install itself relies on finding
site-packages relative to sys.prefix forces the issue - easy_install
(including existing versions of it) needs to just work.
2) Repoint sys.prefix, and require third-party code to understand some
new sys.system_prefix or sys.installed_prefix or equivalent to find the
rest of the base Python installation.
I think that the answer is for third party code not to use any prefix
directly,
but rely on an API such as sysconfig.get_path('purelib'). There's a distutils
version of this - distutils.sysconfig.get_python_lib() - but this is brittle
as
per issue #6087.
Yes, of course. If all code did this, we wouldn't have this problem; we
would just fix sysconfig.py and site.py and all would be well.
3) Attempt the virtualenv approach of trying to fool third-party code
into thinking the virtualenv IS a full Python installation, even though
it's not.
I think #1 is far preferable to #2, and in the long run preferable to #3
even if it involves some short-term pain. But maybe I'm being too
idealistic, and #3 is the pragmatic choice even though it's wrong.
I'm afraid you may be right, at least until people stop using sys.prefix and
sys.exec_prefix directly and call functions instead, which can operate
differently in a virtual environment.
Yup.
Carl
___
Distutils-SIG maillist - Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] pythonv, take two
On 03/31/2011 09:21 AM, Vinay Sajip wrote: Vinay Sajip vinay_sajip at yahoo.co.uk writes: I think the problem is occurring because there's incomplete support for running from a source build. To confirm this, I installed pythonv - and then running pmv.py produces the expected result - an env with a copied Python which allows you to install stuff into the env. I packaged the pythonv I used as a .deb using checkinstall, so that it's uninstallable: http://www.red-dove.com/pythonv_3.3-1_i386.deb and having installed it, I re-ran pmv and did some smoke testing, output is at http://paste.pocoo.org/show/363186/ I know we're not done yet, but at least this shows that the getpath.c code seems to be working (because sys.path seems correct when running the env's copy of Python) and distribute_setup.py installs without errors, followed by easy_installing sample packages setuptools-git, ply, Pygments, Jinja2, SQLAlchemy and coverage ... all seemingly without problems. Of course, this is still using sys.prefix/sys.exec_prefix. Looks great! I can confirm it works when installed. I'll look into the source-build issues. Carl ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] pythonv, take two
Carl Meyer carl at oddbird.net writes:
The logical conclusion of that approach, if you want
full compatibility with third-party code, is to just copy the entire
Python installation to the new location.
We do want that compatibility, don't we? Using a virtual environment should be a
no-brainer, as virtualenv currently is (most of the time). Of course, the
downside of the copying would be around 50MB per virtualenv on Windows, and much
less where symlinking or hardlinking is available. Not ideal, of course, but
disk space is cheap ;-)
So three options as I see it:
1) Add sys.virtual_prefix (or perhaps sys.site_prefix would be a better
name), and require third-party code to use that (or sysconfig) to find
site-packages, if it wants to be virtual-compatible.
But the success of virtualenv is because you can install just about any package
in an environment, and packages don't have to do anything to be virtual
compatible.
2) Repoint sys.prefix, and require third-party code to understand some
new sys.system_prefix or sys.installed_prefix or equivalent to find the
rest of the base Python installation.
I think that the answer is for third party code not to use any prefix directly,
but rely on an API such as sysconfig.get_path('purelib'). There's a distutils
version of this - distutils.sysconfig.get_python_lib() - but this is brittle as
per issue #6087.
3) Attempt the virtualenv approach of trying to fool third-party code
into thinking the virtualenv IS a full Python installation, even though
it's not.
I think #1 is far preferable to #2, and in the long run preferable to #3
even if it involves some short-term pain. But maybe I'm being too
idealistic, and #3 is the pragmatic choice even though it's wrong.
I'm afraid you may be right, at least until people stop using sys.prefix and
sys.exec_prefix directly and call functions instead, which can operate
differently in a virtual environment.
Regards,
Vinay Sajip
___
Distutils-SIG maillist - Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] pythonv, take two
Carl Meyer carl at oddbird.net writes: Hmm, it doesn't seem to be working for me. sys.prefix and sys.exec_prefix are still /usr/local using the python binary in my pmv, and sys.virtual_prefix is not set at all, despite env.cfg being present and looking as it should. Oddly when I run the compiled python binary directly from the checkout, it does set sys.virtual_prefix to point one level above the checkout, despite there being no env.cfg in the vicinity at all. You can see both of these things (in reverse order) here: http://paste.pocoo.org/show/362798/ After testing on a machine with no Python 3.3 installed, I get a similar failure to yours (invalid Python installation due to missing Makefile). However, rhe getpath.c code seems to be working as expected: see http://paste.pocoo.org/show/363159/ for example. The path has been set to reflect the source build where the python was copied from. Of course at this point virtualize() hasn't been called, so virtual_prefix etc. haven't been set, and prefixes are pointing to /usr/local. I think the problem is occurring because there's incomplete support for running from a source build. I'm not sure what to do about this; I think proper support will need changes in multiple other places, which I haven't bottomed out yet but which includes distutils.sysconfig (see issue #6087). Since running from a source build does not set sys.prefix and sys.exec_prefix, either this needs to change, or there needs to be some additional concept like sys.python_home which is available to scripts, or perhaps something else. Of course fixing the source build issue is of use only for core developers, so I'm not sure what appetite there'd be for regularising things for this case. Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] pythonv, take two
Vinay Sajip vinay_sajip at yahoo.co.uk writes: I think the problem is occurring because there's incomplete support for running from a source build. To confirm this, I installed pythonv - and then running pmv.py produces the expected result - an env with a copied Python which allows you to install stuff into the env. I packaged the pythonv I used as a .deb using checkinstall, so that it's uninstallable: http://www.red-dove.com/pythonv_3.3-1_i386.deb and having installed it, I re-ran pmv and did some smoke testing, output is at http://paste.pocoo.org/show/363186/ I know we're not done yet, but at least this shows that the getpath.c code seems to be working (because sys.path seems correct when running the env's copy of Python) and distribute_setup.py installs without errors, followed by easy_installing sample packages setuptools-git, ply, Pygments, Jinja2, SQLAlchemy and coverage ... all seemingly without problems. Of course, this is still using sys.prefix/sys.exec_prefix. Regards, Vinay ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] pythonv, take two
Hi Vinay, On 03/29/2011 07:00 PM, Vinay Sajip wrote: If we switch to sys.virtual_prefix and leave sys.prefix alone, the danger would be that some installer that's using sys.prefix directly, and ignoring sysconfig.py, would try to install stuff to the global Python installation. But I'm not aware of any such installer, and I think we could safely call any installer of Python code that ignores the layout schemes in sysconfig thoroughly broken anyway. Your thoughts? (Or anyone else?) What about the possibility of third-party code which uses sys.prefix/sys.exec_prefix, though? (I mean apart from the installer code.) Right. The question is, for third-party code using sys.prefix/sys.exec_prefix, which value would be the right one, the installation prefix or the virtual prefix? It depends on what they're using it for. I did a quick Google Code Search for sys.prefix, and unfortunately right on the front page I can find examples where either approach we take would be wrong. The majority of uses I found are people copying code from trace.py which is intended to exclude the standard library and system-wide packages from coverage tracing. In this case, leaving sys.prefix pointing to the installation prefix would give the desired result. Also, for instance, here where matplotlib's setup looks for DLLs on Windows, the system prefix would be correct: http://bit.ly/eQSVYc On the other hand, here scons uses sys.prefix to try to find site-packages: http://bit.ly/ik0PwJ If we leave sys.prefix pointing to the installation directory, this would leave scons unable to find its config file in the virtualenv's site-packages directory. Unless they fix their code to use the get_python_lib() function in sysconfig, which is what they ought to be doing instead of duplicating all that logic. So... it seems to me that we're likely to break _some_ third-party code using sys.prefix regardless of what we do here. My instinct says adding sys.virtual_prefix and leaving sys.prefix alone is the better approach, but I'm not very firmly entrenched in that position. On the question of a C-readable configuration file, I've started work on this on the assumption that it's a UTF8-encoded file which contains lines of the form key = value. Having thought about it, since this file is only for bootstrapping the virtualenv anyway, there's no strong case for it to be ConfigParser-compatible. Agreed. I only used ConfigParser initially because that was easiest from Python code :-) I've modified calculate_path in getpath.c to look for the file adjacent to the executable or one level up. If found, we look for a line home = /path/to/source/of/copied/python and, if found, replace argv0_path with the value, then let the existing code take its course. I've modified your virtualize() in site.py to cater for the new config file format: we default the prefix to the parent of the directory containing the executable (hence, the virtualenv root), then set sys.virtual_prefix to this prefix, and optionally disallow the system site-packages from the system path. For now (since [distutils.]sysconfig is not sys.virtual_prefix aware) I continue to set sys.prefix and sys.exec_prefix to sys.virtual_prefix. This sounds good. So as not to cause confusion with existing config files you might have, I changed the config file name to 'env.cfg'. This can be changed to the appropriate bikeshed colour in two places - site.py and getpath.c. Progress seems good. To test: 1. Clone the repo at https://bitbucket.org/vinay.sajip/pythonv 2. In that repo, run ./configure and make 3. Download the pmv.py script at https://gist.github.com/893507 4. Run pmv.py with the python you just built - see the header comment in that script. Note that the pmv.py (Poor Man's virtualize.py) script deliberately runs with a copied rather than a symlinked Python. Hmm, it doesn't seem to be working for me. sys.prefix and sys.exec_prefix are still /usr/local using the python binary in my pmv, and sys.virtual_prefix is not set at all, despite env.cfg being present and looking as it should. Oddly when I run the compiled python binary directly from the checkout, it does set sys.virtual_prefix to point one level above the checkout, despite there being no env.cfg in the vicinity at all. You can see both of these things (in reverse order) here: http://paste.pocoo.org/show/362798/ I'll dig in a bit more and see if I can figure out what's happening. Is there a reason you didn't base your changesets on mine (in the cpythonv repo), and instead apparently copied over a bunch of the changes manually as a diff? I can transplant or copy your work back, or try to do a full merge, but in the long run if we'll both be working on this it seems best if we are easily able to merge our work back and forth. Carl ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] pythonv, take two
Carl Meyer carl at oddbird.net writes: So... it seems to me that we're likely to break _some_ third-party code using sys.prefix regardless of what we do here. My instinct says adding sys.virtual_prefix and leaving sys.prefix alone is the better approach, but I'm not very firmly entrenched in that position. That's problematic, of course (third-party breakage, I mean). It may be that despite using virtual_prefix being a cleaner approach, it may be necessary to alter sys.prefix to ensure compatibility, a la virtualenv. Hmm, it doesn't seem to be working for me. sys.prefix and sys.exec_prefix are still /usr/local using the python binary in my pmv, and sys.virtual_prefix is not set at all, despite env.cfg being present and looking as it should. Oddly when I run the compiled python binary directly from the checkout, it does set sys.virtual_prefix to point one level above the checkout, despite there being no env.cfg in the vicinity at all. You can see both of these things (in reverse order) here: http://paste.pocoo.org/show/362798/ I'll dig in a bit more and see if I can figure out what's happening. It may be that the thing isn't working properly, but it's masked on my system because I have an installed 3.3 in /usr/local (from when I was doing my pip/virtualenv tests). This means that there is a Makefile in a location which would not be there if I'd never installed 3.3 using sudo make install. However, the getpath.c stuff should work - to test this, insert an import pdb; pdb.set_trace() in site.main, and check sys.path. This should contain entries from the virtualenv. The problems are more likely to be in the Python code: the C code always sets sys.prefix and sys.exec_prefix to /usr/local (or whatever configure was invoked with) and the sys.prefix / sys.exec_prefix stuff is done in site.virtualize, as before. I'll try to test this on a machine which never had 3.3 installed - it seems you can't uninstall Python using sudo make uninstall :-( Is there a reason you didn't base your changesets on mine (in the cpythonv repo), and instead apparently copied over a bunch of the changes manually as a diff? I can transplant or copy your work back, or try to do a full merge, but in the long run if we'll both be working on this it seems best if we are easily able to merge our work back and forth. Sorry - I got in a muddle because I've got too many clones around and ran into free disk space issues, and goofed somewhere. Don't bother to do a full merge for now, as I'm not sure if my code's working yet. Since I don't expect to change more than 4 files (getpath.c, site.py, distutils/sysconfig.py and sysconfig.py) I don't think we'll have too much trouble merging in due course, and if/when we can agree that my experiment in pythonv is worth taking further, I'll do the work of merging into cpythonv. For now, though, my changes might be treading on your toes, making merging needlessly annoying. I'll try to test on a Python-clean machine in a day or two, will report back then. Regards, Vinay ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] pythonv, take two
At 01:35 PM 3/30/2011 -0400, Carl Meyer wrote: So... it seems to me that we're likely to break _some_ third-party code using sys.prefix regardless of what we do here. My instinct says adding sys.virtual_prefix and leaving sys.prefix alone is the better approach, but I'm not very firmly entrenched in that position. Looking at it from a software distribution POV, I would say that the virtual prefix is what it should point to, since that means things won't get installed to the wrong place. (Of course, a configuration option could be used to override it, if really necessary.) ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] pythonv, take two
On 03/30/2011 07:18 PM, Vinay Sajip wrote: So... it seems to me that we're likely to break _some_ third-party code using sys.prefix regardless of what we do here. My instinct says adding sys.virtual_prefix and leaving sys.prefix alone is the better approach, but I'm not very firmly entrenched in that position. That's problematic, of course (third-party breakage, I mean). It may be that despite using virtual_prefix being a cleaner approach, it may be necessary to alter sys.prefix to ensure compatibility, a la virtualenv. Well, as I said, I think there will be compatibility problems either way. We are splitting site-packages from the base Python installation, and when third-party code assumes that site-packages and the rest of the base Python installation are both found relative to sys.prefix, neither choice we make can possibly be right. Virtualenv tries to avoid this problem by copying or symlinking in enough stuff from the base Python install in order to convince third parties using sys.prefix that they really are looking at a full Python installation. But that approach is hacky and far from problem-free, and requires keeping up with various things that third-party code might try to look for under sys.prefix (there is at least one current open bug on virtualenv, Tkinter failing in a Windows virtualenv, that is the result of this problem). The logical conclusion of that approach, if you want full compatibility with third-party code, is to just copy the entire Python installation to the new location. So three options as I see it: 1) Add sys.virtual_prefix (or perhaps sys.site_prefix would be a better name), and require third-party code to use that (or sysconfig) to find site-packages, if it wants to be virtual-compatible. 2) Repoint sys.prefix, and require third-party code to understand some new sys.system_prefix or sys.installed_prefix or equivalent to find the rest of the base Python installation. 3) Attempt the virtualenv approach of trying to fool third-party code into thinking the virtualenv IS a full Python installation, even though it's not. I think #1 is far preferable to #2, and in the long run preferable to #3 even if it involves some short-term pain. But maybe I'm being too idealistic, and #3 is the pragmatic choice even though it's wrong. It may be that the thing isn't working properly, but it's masked on my system because I have an installed 3.3 in /usr/local (from when I was doing my pip/virtualenv tests). This means that there is a Makefile in a location which would not be there if I'd never installed 3.3 using sudo make install. However, the getpath.c stuff should work - to test this, insert an import pdb; pdb.set_trace() in site.main, and check sys.path. This should contain entries from the virtualenv. The problems are more likely to be in the Python code: the C code always sets sys.prefix and sys.exec_prefix to /usr/local (or whatever configure was invoked with) and the sys.prefix / sys.exec_prefix stuff is done in site.virtualize, as before. No, if I break early in site.py (or run python -S), sys.path doesn't contain any virtualenv paths. Sorry - I got in a muddle because I've got too many clones around and ran into free disk space issues, and goofed somewhere. Don't bother to do a full merge for now, as I'm not sure if my code's working yet. Since I don't expect to change more than 4 files (getpath.c, site.py, distutils/sysconfig.py and sysconfig.py) I don't think we'll have too much trouble merging in due course, and if/when we can agree that my experiment in pythonv is worth taking further, I'll do the work of merging into cpythonv. For now, though, my changes might be treading on your toes, making merging needlessly annoying. I'll try to test on a Python-clean machine in a day or two, will report back then. Ok, thanks! Carl ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] pythonv, take two
On 03/30/2011 11:23 PM, P.J. Eby wrote: At 01:35 PM 3/30/2011 -0400, Carl Meyer wrote: So... it seems to me that we're likely to break _some_ third-party code using sys.prefix regardless of what we do here. My instinct says adding sys.virtual_prefix and leaving sys.prefix alone is the better approach, but I'm not very firmly entrenched in that position. Looking at it from a software distribution POV, I would say that the virtual prefix is what it should point to, since that means things won't get installed to the wrong place. Indeed. The issue is that from every point of view other than software distribution (code trying to find anything other than site-directories), sys.prefix pointing to the virtualenv is wrong. Unless, like virtualenv, we try to make it just right enough by copying/symlinking things into the virtualenv that otherwise wouldn't need to be there. But this may be moot; I didn't realize until I checked just now that setuptools (well, easy_install) builds its own list of site-dirs based on sys.prefix. It doesn't look like that's used for installation, but it is used for pre-flight checking pth-capability and finding pth files. So if easy_install directly relies on site-directories always being sys.prefix-based, that probably forces the issue. (Of course, a configuration option could be used to override it, if really necessary.) This seems like the worst option of all - then third-party code really would have no idea at all what sys.prefix is supposed to mean, or what can be reliably found there. Carl ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] pythonv, take two
Carl Meyer carl at oddbird.net writes:
Thanks for the additional digging in here. I think your analysis is
right - it actually occurred to me yesterday that this could be the
problem, and I filed a bug to track it here:
3. The fully-reliable fix would be to somehow give the copied binary a
hint where to find the right standard library, and this would involve
adding something to the algorithm in getpath.c. The hint could take the
form of a key in the config file, but I'd really like to avoid fully
parsing the config-file in C and then again in Python later on. The hint
could also be some kind of specially-formatted comment line at the top
of the config file, which would require less C code to find and parse?
Any thoughts on this (or alternative solutions I haven't thought of) are
most welcome.
Hi Carl,
I've been thinking about the C-level configuration some more (cpythonv issue
#6). Suppose we have a virtual env at absolute path testenv, and in
testenv/bin we have a copied, non-installed Python, obtained from absolute
path pythonsrc. Say we provide the location of the source in the
configuration, and that this is read in getpath.c. Logically, it should just
replace the argv0_path value, and if the subsequent logic is unchanged, at the
point that site.py's main() runs, you would have (on Linux i686):
sys.prefix, sys.exec_prefix would have values set from the configure script
('/usr/local' by default).
sys.path would be ['/usr/local/lib/python33.zip', 'pythonsrc/Lib',
'pythonsrc/Lib/plat-linux2', 'pythonsrc/build/lib.linux-i686-3.3']
which seems OK.
If a virtual environment is in absolute path testenv, then after the call to
virtualize() in site.py, we would have sys.prefix and sys.exec_prefix both set
to testenv. Since sys.executable still points to testenv/bin, the code in
sysconfig.py appears not to be doing the right thing, e.g.
sysconfig._PROJECT_BASE would be testenv/bin, which seems wrong.
So it seems as if site.py and/or sysconfig.py might need further changes, as
well as changes to getpath.c. What do you think?
Regards,
Vinay Sajip
___
Distutils-SIG maillist - Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] pythonv, take two
Hi Vinay,
On 03/29/2011 09:45 AM, Vinay Sajip wrote:
I've been thinking about the C-level configuration some more (cpythonv issue
#6). Suppose we have a virtual env at absolute path testenv, and in
testenv/bin we have a copied, non-installed Python, obtained from absolute
path pythonsrc. Say we provide the location of the source in the
configuration, and that this is read in getpath.c. Logically, it should just
replace the argv0_path value, and if the subsequent logic is unchanged, at the
point that site.py's main() runs, you would have (on Linux i686):
sys.prefix, sys.exec_prefix would have values set from the configure script
('/usr/local' by default).
sys.path would be ['/usr/local/lib/python33.zip', 'pythonsrc/Lib',
'pythonsrc/Lib/plat-linux2', 'pythonsrc/build/lib.linux-i686-3.3']
which seems OK.
If a virtual environment is in absolute path testenv, then after the call to
virtualize() in site.py, we would have sys.prefix and sys.exec_prefix both set
to testenv. Since sys.executable still points to testenv/bin, the code in
sysconfig.py appears not to be doing the right thing, e.g.
sysconfig._PROJECT_BASE would be testenv/bin, which seems wrong.
So it seems as if site.py and/or sysconfig.py might need further changes, as
well as changes to getpath.c. What do you think?
In general, I think the copied binary case should be as similar as
possible to the symlinked binary case. Python doesn't dereference
symlinks in setting sys.executable, so in either case sys.executable
will point to the virtual environment's binary, which I think is clearly
what we want (think code using sys.executable to decide how to re-exec
itself).
The changes I made to sysconfig thus far were the minimum necessary to
be able to successfully install stuff in a pythonv virtualenv. On a
brief glance, it looks to me like you are right; we'll need to modify
the setting of _PROJECT_BASE in sysconfig.py to be virtual-aware (this
is part -- maybe all -- of what's needed to fix issue #1).
Also, see issue #10 - I'm wavering on whether changing sys.prefix and
sys.exec_prefix to point to the virtualenv is actually the right
approach. We're already forced to modify sysconfig.py to be
virtual-aware in order to avoid having to hackily copy/symlink bits of
the stdlib and include directories and the Makefile and config.h and
whatnot into the virtualenv. Once we've taken the step of making
sysconfig virtual-aware, it seems to me that perhaps sys.prefix is
better left pointing to the actual Python installation, and we add a
sys.virtual_prefix or some such to tell site.py and sysconfig.py where
virtualized stuff belongs.
There are a variety of possible meanings of sys.prefix, and the only one
we really want to modify is where should I install third-party
packages? Looking at sys.prefix references in the stdlib, I see them in
trace, tkinter, and idlelib. It appears to me that in all of those
cases, having sys.prefix pointing to the virtualenv would be wrong.
If we switch to sys.virtual_prefix and leave sys.prefix alone, the
danger would be that some installer that's using sys.prefix directly,
and ignoring sysconfig.py, would try to install stuff to the global
Python installation. But I'm not aware of any such installer, and I
think we could safely call any installer of Python code that ignores the
layout schemes in sysconfig thoroughly broken anyway.
Your thoughts? (Or anyone else?)
Carl
P.S. The elephant in the room here is the fact that if we want pythonv
to be backwards-compatible with distutils/distribute, which we do, we
can't get away with just modifying sysconfig.py, we also have to modify
the old distutils/sysconfig.py. In general distutils is frozen; I'm
hoping that Tarek will let me get away with a few changes for pythonv if
they aren't changing APIs at all, just modifying some install-scheme
paths to be virtual-aware.
___
Distutils-SIG maillist - Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] pythonv, take two
Carl Meyer carl at oddbird.net writes: In general, I think the copied binary case should be as similar as possible to the symlinked binary case. Python doesn't dereference symlinks in setting sys.executable, so in either case sys.executable will point to the virtual environment's binary, which I think is clearly what we want (think code using sys.executable to decide how to re-exec itself). I agree. The changes I made to sysconfig thus far were the minimum necessary to be able to successfully install stuff in a pythonv virtualenv. On a brief glance, it looks to me like you are right; we'll need to modify the setting of _PROJECT_BASE in sysconfig.py to be virtual-aware (this is part -- maybe all -- of what's needed to fix issue #1). Also, see issue #10 - I'm wavering on whether changing sys.prefix and sys.exec_prefix to point to the virtualenv is actually the right approach. We're already forced to modify sysconfig.py to be virtual-aware in order to avoid having to hackily copy/symlink bits of the stdlib and include directories and the Makefile and config.h and whatnot into the virtualenv. Once we've taken the step of making sysconfig virtual-aware, it seems to me that perhaps sys.prefix is better left pointing to the actual Python installation, and we add a sys.virtual_prefix or some such to tell site.py and sysconfig.py where virtualized stuff belongs. This sounds better, too. If we switch to sys.virtual_prefix and leave sys.prefix alone, the danger would be that some installer that's using sys.prefix directly, and ignoring sysconfig.py, would try to install stuff to the global Python installation. But I'm not aware of any such installer, and I think we could safely call any installer of Python code that ignores the layout schemes in sysconfig thoroughly broken anyway. Your thoughts? (Or anyone else?) What about the possibility of third-party code which uses sys.prefix/sys.exec_prefix, though? (I mean apart from the installer code.) P.S. The elephant in the room here is the fact that if we want pythonv to be backwards-compatible with distutils/distribute, which we do, we can't get away with just modifying sysconfig.py, we also have to modify the old distutils/sysconfig.py. In general distutils is frozen; I'm hoping that Tarek will let me get away with a few changes for pythonv if they aren't changing APIs at all, just modifying some install-scheme paths to be virtual-aware. Here's hoping :-) On the question of a C-readable configuration file, I've started work on this on the assumption that it's a UTF8-encoded file which contains lines of the form key = value. Having thought about it, since this file is only for bootstrapping the virtualenv anyway, there's no strong case for it to be ConfigParser-compatible. I've modified calculate_path in getpath.c to look for the file adjacent to the executable or one level up. If found, we look for a line home = /path/to/source/of/copied/python and, if found, replace argv0_path with the value, then let the existing code take its course. I've modified your virtualize() in site.py to cater for the new config file format: we default the prefix to the parent of the directory containing the executable (hence, the virtualenv root), then set sys.virtual_prefix to this prefix, and optionally disallow the system site-packages from the system path. For now (since [distutils.]sysconfig is not sys.virtual_prefix aware) I continue to set sys.prefix and sys.exec_prefix to sys.virtual_prefix. So as not to cause confusion with existing config files you might have, I changed the config file name to 'env.cfg'. This can be changed to the appropriate bikeshed colour in two places - site.py and getpath.c. Progress seems good. To test: 1. Clone the repo at https://bitbucket.org/vinay.sajip/pythonv 2. In that repo, run ./configure and make 3. Download the pmv.py script at https://gist.github.com/893507 4. Run pmv.py with the python you just built - see the header comment in that script. Note that the pmv.py (Poor Man's virtualize.py) script deliberately runs with a copied rather than a symlinked Python. Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] pythonv, take two
On Fri, Mar 18, 2011 at 6:35 PM, Carl Meyer c...@oddbird.net wrote: On 03/18/2011 05:24 PM, Tres Seaver wrote: Could the config file contain an optional hint for finding the right stdlib in cases where the binary copy had been made? I realize that parsing a config file *without* the stdlib is painful: perhaps looking for a line starting with 'stdlib =' would be enough? Yes, this was one possible solution I was considering. Two things bother me about it: 1. It seems... tempting but ill-advised to make it a fake ConfigParser option if in the place where I use it, I'm not actually parsing the file with full ConfigParser semantics. It might work, but then someone tries to get clever with a [DEFAULT] section, or interpolation, or who knows what else, and it breaks mysteriously. That's why I was leaning towards a specially-tagged comment at the top of the file. Ugly, but at least not pretending to be something it isn't. Don't call it ConfigParser format. Call it ini format, or whatever. http://bit.ly/hNVmhz Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] pythonv, take two
Hi Vinay, Thanks for the additional digging in here. I think your analysis is right - it actually occurred to me yesterday that this could be the problem, and I filed a bug to track it here: https://bitbucket.org/carljm/cpythonv/issue/6/if-binary-is-copied-prefix-finding-could The issue is finding the initial sys.prefix and thus the standard library, which happens in the search_for_prefix function Modules/getpath.c. The algorithm used is this (presuming no PYTHONHOME): 1. Step up the tree from the (symlinks-dereferenced) binary location looking for anything that appears to be the standard library. 2. Fall back to the location hardcoded in the binary from the --prefix option to ./configure. With a symlink it's fully reliable, because the symlink is dereferenced so the stdlib is found just as it normally would be. With a copy of the binary, there are two situations where it can break: A. The Python installation we are copying from has an incorrect hardcoded prefix in the binary, and is relying on the tree-stepping to find its standard library. In this case, the hardcoded prefix will be used because the tree-stepping won't find anything. I think this is the situation you are seeing: I think your binary is compiling with a prefix of /usr/local, and so it falls back on that prefix when it can't find anything by tree-stepping up through /home. If you didn't actually have a Python 3.3 installation in /usr/local, it would break more loudly due to not finding any stdlib at all. I think you'll find that it works for you if you compile with an explicit --prefix and then actually install to that prefix, then copy that binary and make an env with it. B. There is _another_ Python standard library (from the same version of Python, on *nix - the Windows stdlib isn't in a version-specific directory) up the hierarchy from the location of our virtualenv. In this case, tree-stepping might find the wrong standard library and use it. I think this case should be quite rare. It's most problematic on Windows, but that's also where it shouldn't really happen, since Python is installed self-contained and I can't imagine why you'd make a virtualenv for one Python version _inside_ the installation directory of another Python version. In Linux I guess somebody might try to make a virtualenv, with a copied binary, somewhere in /usr/local, using some Python _other_ than the one installed in /usr, but with the same major/minor version. A very edge case, but possible. So, possibilities I see for addressing this: 1. Decide that in real cases of real Python installations, it's so unlikely to happen that we won't worry about it. I think it's possible that this is acceptable; the biggest practical problem is likely to be people trying to test this out during PEP review from a not-installed checkout, just like you did. We'd have to be careful to instruct people that it doesn't work that way, and might also want to add a check in the env-creation script to verify that the created env works properly, and if it doesn't give them some clue why not. 2. Decide that we just don't support copied binaries, only symlinked ones. Apparently (I am Windows-ignorant) recent Windows versions do support symlinks? So this might only involve dropping support for old Windows'? How important is it for a new feature like this to fully support all operating systems that Python supports? We could also not expose the copy-binary option to the user, but fall back to it if we have no symlinks; which ends up being option (1) but trying to narrow even more the potential breakage cases. 3. The fully-reliable fix would be to somehow give the copied binary a hint where to find the right standard library, and this would involve adding something to the algorithm in getpath.c. The hint could take the form of a key in the config file, but I'd really like to avoid fully parsing the config-file in C and then again in Python later on. The hint could also be some kind of specially-formatted comment line at the top of the config file, which would require less C code to find and parse? Any thoughts on this (or alternative solutions I haven't thought of) are most welcome. Carl ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] pythonv, take two
At 02:44 PM 3/18/2011 -0400, Carl Meyer wrote: Apparently (I am Windows-ignorant) recent Windows versions do support symlinks? Technically, some Windows filesystems can support this. In practice, no user-visible tools actually support making or using them sanely, AFAIK. So, I suggest promoting symlinks as the standard way, with binary-copying being a Windows-only workaround. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] pythonv, take two
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 03/18/2011 03:20 PM, P.J. Eby wrote: At 02:44 PM 3/18/2011 -0400, Carl Meyer wrote: Apparently (I am Windows-ignorant) recent Windows versions do support symlinks? Technically, some Windows filesystems can support this. In practice, no user-visible tools actually support making or using them sanely, AFAIK. So, I suggest promoting symlinks as the standard way, with binary-copying being a Windows-only workaround. Could the config file contain an optional hint for finding the right stdlib in cases where the binary copy had been made? I realize that parsing a config file *without* the stdlib is painful: perhaps looking for a line starting with 'stdlib =' would be enough? Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk2DzZUACgkQ+gerLs4ltQ6YtgCfcXlvVYLfwaLLfbAkF9YzOzog 9ekAnAhVygL2APAWR+Cd5h5ePEYblbBN =wazt -END PGP SIGNATURE- ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] pythonv, take two
On 03/18/2011 05:24 PM, Tres Seaver wrote: Could the config file contain an optional hint for finding the right stdlib in cases where the binary copy had been made? I realize that parsing a config file *without* the stdlib is painful: perhaps looking for a line starting with 'stdlib =' would be enough? Yes, this was one possible solution I was considering. Two things bother me about it: 1. It seems... tempting but ill-advised to make it a fake ConfigParser option if in the place where I use it, I'm not actually parsing the file with full ConfigParser semantics. It might work, but then someone tries to get clever with a [DEFAULT] section, or interpolation, or who knows what else, and it breaks mysteriously. That's why I was leaning towards a specially-tagged comment at the top of the file. Ugly, but at least not pretending to be something it isn't. 2. I still have to duplicate in C and Python code the logic for finding the config file in the first place. Or I guess just move it to C and stash the location in sys or somewhere the Python code can get to it later. I'd been hoping to avoid this, but oh well... Carl ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] pythonv, take two
1. It seems... tempting but ill-advised to make it a fake ConfigParser option if in the place where I use it, I'm not actually parsing the file with full ConfigParser semantics. It might work, but then someone tries to get clever with a [DEFAULT] section, or interpolation, or who knows what else, and it breaks mysteriously. It’s still time to define the file format as RawConfigParser-style, without interpolation. 2. I still have to duplicate in C and Python code the logic for finding the config file in the first place. Or I guess just move it to C and stash the location in sys or somewhere the Python code can get to it later. I'd been hoping to avoid this, but oh well... *hands a courage cookie* I have to say the proof of concept in pure Python was impressive. Too bad it’s not enough :) Cheers ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] pythonv, take two
On 03/18/2011 07:22 PM, Éric Araujo wrote: 1. It seems... tempting but ill-advised to make it a fake ConfigParser option if in the place where I use it, I'm not actually parsing the file with full ConfigParser semantics. It might work, but then someone tries to get clever with a [DEFAULT] section, or interpolation, or who knows what else, and it breaks mysteriously. It’s still time to define the file format as RawConfigParser-style, without interpolation. Indeed. And I guess there's little to no reason why someone would try to define a default section, or use multiline syntax, or anything else like that. Perhaps this would be robust enough. 2. I still have to duplicate in C and Python code the logic for finding the config file in the first place. Or I guess just move it to C and stash the location in sys or somewhere the Python code can get to it later. I'd been hoping to avoid this, but oh well... *hands a courage cookie* I have to say the proof of concept in pure Python was impressive. Too bad it’s not enough :) Courage cookie accepted ;) Carl ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] pythonv, take two
Hi Carl, So, possibilities I see for addressing this: 1. Decide that in real cases of real Python installations, it's so unlikely to happen that we won't worry about it. I think it's possible that this is acceptable; the biggest practical problem is likely to be people trying to test this out during PEP review from a not-installed checkout, just like you did. We'd have to be careful to instruct people that it doesn't work that way, and might also want to add a check in the env-creation script to verify that the created env works properly, and if it doesn't give them some clue why not. Yes, I think this is important to do, or at least to have -v output show which stdlib is being used. 2. Decide that we just don't support copied binaries, only symlinked ones. Apparently (I am Windows-ignorant) recent Windows versions do support symlinks? So this might only involve dropping support for old Windows'? How important is it for a new feature like this to fully support all operating systems that Python supports? We could also not expose the copy-binary option to the user, but fall back to it if we have no symlinks; which ends up being option (1) but trying to narrow even more the potential breakage cases. Well, Windows XP is still pretty common and likely to remain so for some time. On Windows XP, symlinks don't work, but confusingly os.symlink exists on recent Pythons (certainly 3.2). I had to make a patch in my virtualenv fork to handle this in copyfile: on XP, os.symlink raises NotImplementedError because the underlying Windows API function is missing. So, even if it's just for Windows, I think copying will need to be supported, but perhaps only from Pythons whose sys.prefix is correctly set. 3. The fully-reliable fix would be to somehow give the copied binary a hint where to find the right standard library, and this would involve adding something to the algorithm in getpath.c. The hint could take the form of a key in the config file, but I'd really like to avoid fully parsing the config-file in C and then again in Python later on. The hint could also be some kind of specially-formatted comment line at the top of the config file, which would require less C code to find and parse? Any thoughts on this (or alternative solutions I haven't thought of) are most welcome. I know that ConfigParser format files are the first thing one thinks of, but perhaps a simpler C-parseable format deserves further consideration. Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] pythonv, take two
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 03/18/2011 08:01 PM, Vinay Sajip wrote: Hi Carl, So, possibilities I see for addressing this: 1. Decide that in real cases of real Python installations, it's so unlikely to happen that we won't worry about it. I think it's possible that this is acceptable; the biggest practical problem is likely to be people trying to test this out during PEP review from a not-installed checkout, just like you did. We'd have to be careful to instruct people that it doesn't work that way, and might also want to add a check in the env-creation script to verify that the created env works properly, and if it doesn't give them some clue why not. Yes, I think this is important to do, or at least to have -v output show which stdlib is being used. 2. Decide that we just don't support copied binaries, only symlinked ones. Apparently (I am Windows-ignorant) recent Windows versions do support symlinks? So this might only involve dropping support for old Windows'? How important is it for a new feature like this to fully support all operating systems that Python supports? We could also not expose the copy-binary option to the user, but fall back to it if we have no symlinks; which ends up being option (1) but trying to narrow even more the potential breakage cases. Well, Windows XP is still pretty common and likely to remain so for some time. On Windows XP, symlinks don't work, but confusingly os.symlink exists on recent Pythons (certainly 3.2). I had to make a patch in my virtualenv fork to handle this in copyfile: on XP, os.symlink raises NotImplementedError because the underlying Windows API function is missing. So, even if it's just for Windows, I think copying will need to be supported, but perhaps only from Pythons whose sys.prefix is correctly set. 3. The fully-reliable fix would be to somehow give the copied binary a hint where to find the right standard library, and this would involve adding something to the algorithm in getpath.c. The hint could take the form of a key in the config file, but I'd really like to avoid fully parsing the config-file in C and then again in Python later on. The hint could also be some kind of specially-formatted comment line at the top of the config file, which would require less C code to find and parse? Any thoughts on this (or alternative solutions I haven't thought of) are most welcome. I know that ConfigParser format files are the first thing one thinks of, but perhaps a simpler C-parseable format deserves further consideration. One possibility is to make the config file format just a sequence of long-form command-line options, one per line, with the leading '--' stripped. This option would involve inventing a '--use-prefix' option (bikeshedders, start your sprayguns ;), but would make parsing it a mostly trivial exercise. Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk2D9bgACgkQ+gerLs4ltQ7VwwCfQku+E1AnGzAYTG6blRgiq6IF n8MAoMGYbjS/E8+2l2LsvhYO2BhLGtEb =mA1G -END PGP SIGNATURE- ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] pythonv, take two
Carl Meyer carl at oddbird.net writes: work well. Please try it and tell me what I missed! To try it out, create a directory somewhere with a bin/ subdirectory, symlink or copy (it works either way) the python3 binary from an install of the pythonv2 branch into bin/, and create lib/python3.3/site-packages It seems to work with a symlink but not with a copy. With an empty pythonv.conf file in ~/projects/vptest: vinay@eta-natty:~/projects/vptest/bin$ ln -s ~/tools/cpythonv/python vinay@eta-natty:~/projects/vptest/bin$ ./python Python 3.3a0 (pythonv2:e56f05883ceb, Mar 17 2011, 09:07:29) [GCC 4.5.2] on linux2 Type help, copyright, credits or license for more information. import sys sys.path ['', '/usr/local/lib/python33.zip', '/home/vinay/tools/cpythonv/Lib', '/home/vinay/tools/cpythonv/Lib/plat-linux2', '/home/vinay/tools/cpythonv/build/lib.linux-i686-3.3', '/home/vinay/projects/vptest/lib/python3.3/site-packages'] vinay@eta-natty:~/projects/vptest/bin$ rm python vinay@eta-natty:~/projects/vptest/bin$ cp ~/tools/cpythonv/python . vinay@eta-natty:~/projects/vptest/bin$ ./python Python 3.3a0 (pythonv2:e56f05883ceb, Mar 17 2011, 09:07:29) [GCC 4.5.2] on linux2 Type help, copyright, credits or license for more information. import sys sys.path ['', '/usr/local/lib/python33.zip', '/usr/local/lib/python3.3', '/usr/local/lib/python3.3/plat-linux2', '/usr/local/lib/python3.3/lib-dynload', '/usr/local/lib/python3.3/site-packages'] Still, very promising! Nice work. Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] pythonv, take two
Hi Vinay, On 03/17/2011 06:25 AM, Vinay Sajip wrote: It seems to work with a symlink but not with a copy. With an empty pythonv.conf file in ~/projects/vptest: vinay@eta-natty:~/projects/vptest/bin$ ln -s ~/tools/cpythonv/python vinay@eta-natty:~/projects/vptest/bin$ ./python Python 3.3a0 (pythonv2:e56f05883ceb, Mar 17 2011, 09:07:29) [GCC 4.5.2] on linux2 Type help, copyright, credits or license for more information. import sys sys.path ['', '/usr/local/lib/python33.zip', '/home/vinay/tools/cpythonv/Lib', '/home/vinay/tools/cpythonv/Lib/plat-linux2', '/home/vinay/tools/cpythonv/build/lib.linux-i686-3.3', '/home/vinay/projects/vptest/lib/python3.3/site-packages'] vinay@eta-natty:~/projects/vptest/bin$ rm python vinay@eta-natty:~/projects/vptest/bin$ cp ~/tools/cpythonv/python . vinay@eta-natty:~/projects/vptest/bin$ ./python Python 3.3a0 (pythonv2:e56f05883ceb, Mar 17 2011, 09:07:29) [GCC 4.5.2] on linux2 Type help, copyright, credits or license for more information. import sys sys.path ['', '/usr/local/lib/python33.zip', '/usr/local/lib/python3.3', '/usr/local/lib/python3.3/plat-linux2', '/usr/local/lib/python3.3/lib-dynload', '/usr/local/lib/python3.3/site-packages'] This is odd, as the same setup works fine for me with a copied binary (I'm on Ubuntu 10.10). If you're willing to do any more debugging on this, here's what would be helpful: 1. Check the values of sys.executable, sys.prefix, and sys.exec_prefix. 2. Stick a pdb.set_trace() into site.py near the top of the virtualize() method and step through it, observing where something goes awry. In particular: does it find pythonv.conf? Does it modify PREFIXES? Thanks for testing it out! Carl ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] pythonv, take two
On 03/16/2011 11:04 PM, Barry Warsaw wrote: +1. Time for a PEP? Working on a draft PEP. I'll push it to bitbucket to make collaboration easier - then the next step would be to present the draft on python-ideas, if I'm reading PEP 2 correctly? Carl ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] pythonv, take two
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 03/17/2011 02:50 PM, Carl Meyer wrote: On 03/16/2011 11:04 PM, Barry Warsaw wrote: +1. Time for a PEP? Working on a draft PEP. I'll push it to bitbucket to make collaboration easier - then the next step would be to present the draft on python-ideas, if I'm reading PEP 2 correctly? Correct. Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk2CXVgACgkQ+gerLs4ltQ7UhwCdHAa2pk5ia3Ls9w4O6SADdeJ7 ehgAnibKo+/zwJ7aydUAYYbuk/xkmvxk =K0bJ -END PGP SIGNATURE- ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] pythonv, take two
On Mar 17, 2011, at 02:50 PM, Carl Meyer wrote: On 03/16/2011 11:04 PM, Barry Warsaw wrote: +1. Time for a PEP? Working on a draft PEP. I'll push it to bitbucket to make collaboration easier - then the next step would be to present the draft on python-ideas, if I'm reading PEP 2 correctly? I dunno. I tend to think python-ideas are where ideas go to die. ;) I think we have general consensus that this is a good idea, and we just need to hash out the details. So I personally think python-dev is fine at this point. (Has nothing to do with me not actually being subscribed to python-ideas. Nope, not at all. :) -Barry signature.asc Description: PGP signature ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] pythonv, take two
Carl Meyer carl at oddbird.net writes: This is odd, as the same setup works fine for me with a copied binary (I'm on Ubuntu 10.10). If you're willing to do any more debugging on this, here's what would be helpful: I think I know what the problem is: the python executable checks to see where it was run from. If it looks as if it was run from a source build, it looks for site.py in the Lib folder relative to the executable; otherwise it looks for site.py in sys.prefix/lib/pythonX.Y. I've copied ~/tools/cpythonv/python to ~/projects/vptest/bin, and here's what happens: vinay@eta-natty:~/projects/vptest/bin$ ~/tools/cpythonv/python Python 3.3a0 (pythonv2:e56f05883ceb, Mar 17 2011, 09:07:29) [GCC 4.5.2] on linux2 Type help, copyright, credits or license for more information. import sys; sys.prefix '/usr/local' sys.path ['', '/usr/local/lib/python33.zip', '/home/vinay/tools/cpythonv/Lib', '/home/vinay/tools/cpythonv/Lib/plat-linux2', '/home/vinay/tools/cpythonv/build/lib.linux-i686-3.3', '/usr/local/lib/python3.3/site-packages'] vinay@eta-natty:~/projects/vptest/bin$ ./python Python 3.3a0 (pythonv2:e56f05883ceb, Mar 17 2011, 09:07:29) [GCC 4.5.2] on linux2 Type help, copyright, credits or license for more information. import sys; sys.prefix '/usr/local' sys.path ['', '/usr/local/lib/python33.zip', '/usr/local/lib/python3.3', '/usr/local/lib/python3.3/plat-linux2', '/usr/local/lib/python3.3/lib-dynload', '/usr/local/lib/python3.3/site-packages'] Notice how the sys.prefix is the same in both cases, but the path is different. Your site.py is never run with the copy, because it gets the site.py from /usr/local/lib/python3.3: vinay@eta-natty:~/projects/vptest/bin$ ~/tools/cpythonv/python Python 3.3a0 (pythonv2:e56f05883ceb, Mar 17 2011, 09:07:29) [GCC 4.5.2] on linux2 Type help, copyright, credits or license for more information. import site; site.__file__ '/home/vinay/tools/cpythonv/Lib/site.py' vinay@eta-natty:~/projects/vptest/bin$ ./python Python 3.3a0 (pythonv2:e56f05883ceb, Mar 17 2011, 09:07:29) [GCC 4.5.2] on linux2 Type help, copyright, credits or license for more information. import site; site.__file__ '/usr/local/lib/python3.3/site.py' I suppose this won't be a problem when the actual Python 3.3 is installed - the virtualizing site.py will be in /usr/lib/python3.3 or /usr/local/lib/python3.3. On this system I have a local build of Python made from the official default branch, which is installed to /usr/local/lib since the system Python is 3.2 (Ubuntu 11.04 - Natty). Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] pythonv, take two
Vinay Sajip vinay_sajip at yahoo.co.uk writes: On this system I have a local build of Python made from the official default branch, which is installed to /usr/local/lib since the system Python is 3.2 (Ubuntu 11.04 - Natty). Sorry, I meant to say that the system Python is 2.7 (in /usr). I also have an official Ubuntu build of 3.2 in /usr, but the source build installs by default to /usr/local. Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] pythonv, take two
On Mar 16, 2011, at 10:33 PM, Carl Meyer wrote: I've pushed this symlink/copy binary approach to the pythonv2 branch at http://bitbucket.org/carljm/cpythonv. I was a bit shocked at how easy it turned out to be: just a few lines in site.py and the same changes to sysconfig.py as previously. No changes in C code needed at all. It was so easy I'm wondering what I must have missed, but everything seems to work well. Please try it and tell me what I missed! Seems to work for me (on Linux at least :)! Really brilliant work, Carl. To try it out, create a directory somewhere with a bin/ subdirectory, symlink or copy (it works either way) the python3 binary from an install of the pythonv2 branch into bin/, and create lib/python3.3/site-packages in the virtualenv. Create pythonv.conf in the root of the virtualenv - it can be empty if you want full isolation; if you want the system site-packages too you can add a [pythonv] section with include-system-site=True (it's a ConfigParser-style file). Start up your binary and check sys.path. You can download distribute and python setup.py install that into your virtualenv and then easy_install more stuff into your env. At this point, unless the lack of LD_LIBRARY_PATH is actually a blocker, I can't see any reason not to go with this approach instead. +1. Time for a PEP? -Barry signature.asc Description: PGP signature ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
