Hi Vinay, Thanks for the additional digging in here. I think your analysis is right - it actually occurred to me yesterday that this could be the problem, and I filed a bug to track it here: https://bitbucket.org/carljm/cpythonv/issue/6/if-binary-is-copied-prefix-finding-could
The issue is finding the initial sys.prefix and thus the standard library, which happens in the search_for_prefix function Modules/getpath.c. The algorithm used is this (presuming no PYTHONHOME): 1. Step up the tree from the (symlinks-dereferenced) binary location looking for anything that appears to be the standard library. 2. Fall back to the location hardcoded in the binary from the --prefix option to ./configure. With a symlink it's fully reliable, because the symlink is dereferenced so the stdlib is found just as it normally would be. With a copy of the binary, there are two situations where it can break: A. The Python installation we are copying from has an incorrect hardcoded prefix in the binary, and is relying on the tree-stepping to find its standard library. In this case, the hardcoded prefix will be used because the tree-stepping won't find anything. I think this is the situation you are seeing: I think your binary is compiling with a prefix of "/usr/local", and so it falls back on that prefix when it can't find anything by tree-stepping up through /home. If you didn't actually have a Python 3.3 installation in /usr/local, it would break more loudly due to not finding any stdlib at all. I think you'll find that it works for you if you compile with an explicit --prefix and then actually install to that prefix, then copy that binary and make an env with it. B. There is _another_ Python standard library (from the same version of Python, on *nix - the Windows stdlib isn't in a version-specific directory) up the hierarchy from the location of our virtualenv. In this case, tree-stepping might find the wrong standard library and use it. I think this case should be quite rare. It's most problematic on Windows, but that's also where it shouldn't really happen, since Python is installed self-contained and I can't imagine why you'd make a virtualenv for one Python version _inside_ the installation directory of another Python version. In Linux I guess somebody might try to make a virtualenv, with a copied binary, somewhere in /usr/local, using some Python _other_ than the one installed in /usr, but with the same major/minor version. A very edge case, but possible. So, possibilities I see for addressing this: 1. Decide that in real cases of real Python installations, it's so unlikely to happen that we won't worry about it. I think it's possible that this is acceptable; the biggest practical problem is likely to be people trying to test this out during PEP review from a not-installed checkout, just like you did. We'd have to be careful to instruct people that it doesn't work that way, and might also want to add a check in the env-creation script to verify that the created env works properly, and if it doesn't give them some clue why not. 2. Decide that we just don't support copied binaries, only symlinked ones. Apparently (I am Windows-ignorant) recent Windows versions do support symlinks? So this might only involve dropping support for old Windows'? How important is it for a new feature like this to fully support all operating systems that Python supports? We could also not expose the copy-binary option to the user, but fall back to it if we have no symlinks; which ends up being option (1) but trying to narrow even more the potential breakage cases. 3. The fully-reliable fix would be to somehow give the copied binary a hint where to find the right standard library, and this would involve adding something to the algorithm in getpath.c. The hint could take the form of a key in the config file, but I'd really like to avoid fully parsing the config-file in C and then again in Python later on. The hint could also be some kind of specially-formatted comment line at the top of the config file, which would require less C code to find and parse? Any thoughts on this (or alternative solutions I haven't thought of) are most welcome. Carl _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
