[issue22490] Using realpath for __PYVENV_LAUNCHER__ makes Homebrew installs fragile

[Jason R. Coombs]

Jason R. Coombs  added the comment:

I've filed [Homebrew/brew#7204](https://github.com/Homebrew/brew/issues/7204) 
to track the testing/validation of this change against Homebrew.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22490] Using realpath for __PYVENV_LAUNCHER__ makes Homebrew installs fragile

[Jason R. Coombs]

Change by Jason R. Coombs :


--
versions: +Python 3.9

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22490] Using realpath for __PYVENV_LAUNCHER__ makes Homebrew installs fragile

[Jason R. Coombs]

Change by Jason R. Coombs :


--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22490] Using realpath for __PYVENV_LAUNCHER__ makes Homebrew installs fragile

[Jason R. Coombs]

Jason R. Coombs  added the comment:


New changeset 5765acaf64cc2c52ce8a35de9407faddf6885598 by Jason R. Coombs in 
branch '3.7':
[3.7] bpo-22490: Remove __PYVENV_LAUNCHER__ from environment during launch 
(GH-9516) (GH-19111)
https://github.com/python/cpython/commit/5765acaf64cc2c52ce8a35de9407faddf6885598


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22490] Using realpath for __PYVENV_LAUNCHER__ makes Homebrew installs fragile

[Jason R. Coombs]

Jason R. Coombs  added the comment:


New changeset c959fa9353b92ce95dd7fe3f25fe65bacbe22338 by Miss Islington (bot) 
in branch '3.8':
bpo-22490: Remove __PYVENV_LAUNCHER__ from environment during launch (GH-9516) 
(GH-19110)
https://github.com/python/cpython/commit/c959fa9353b92ce95dd7fe3f25fe65bacbe22338


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22490] Using realpath for __PYVENV_LAUNCHER__ makes Homebrew installs fragile

[Jason R. Coombs]

Change by Jason R. Coombs :


--
pull_requests: +18472
pull_request: https://github.com/python/cpython/pull/19111

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22490] Using realpath for __PYVENV_LAUNCHER__ makes Homebrew installs fragile

[Jason R. Coombs]

Jason R. Coombs  added the comment:


New changeset 044cf94f610e831464a69a8e713dad89878824ce by Ronald Oussoren in 
branch 'master':
bpo-22490: Remove __PYVENV_LAUNCHER__ from environment during launch (GH-9516)
https://github.com/python/cpython/commit/044cf94f610e831464a69a8e713dad89878824ce


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22490] Using realpath for __PYVENV_LAUNCHER__ makes Homebrew installs fragile

[miss-islington]

Change by miss-islington :


--
nosy: +miss-islington
nosy_count: 8.0 -> 9.0
pull_requests: +18471
pull_request: https://github.com/python/cpython/pull/19110

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22490] Using realpath for __PYVENV_LAUNCHER__ makes Homebrew installs fragile

[Cheryl Sabella]

Cheryl Sabella  added the comment:

It looks like @ronaldoussoren's pull request was ready to go pending a test 
with Brew.  The PR itself needs a rebase, but is anyone able to finish the 
testing on this for it to be merged?  Thanks!

--
nosy: +cheryl.sabella
versions: +Python 3.7, Python 3.8 -Python 3.4, Python 3.5, Python 3.6

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22490] Using realpath for __PYVENV_LAUNCHER__ makes Homebrew installs fragile

[Anders Hovmöller]

Anders Hovmöller  added the comment:

I just discovered this ticket again and see that it's stuck! 

I have read through the thread but it's still a bit unclear what would be 
required to test this with homebrew like Guido says is needed for this to go 
forward. Is there anyone who can explain or better yet test?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22490] Using realpath for __PYVENV_LAUNCHER__ makes Homebrew installs fragile

[Guido van Rossum]

Guido van Rossum  added the comment:

Thanks, I have confirmed that this reproduces our issue and that your PR fixes 
it. I guess at this point we need someone to test with Brew.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22490] Using realpath for __PYVENV_LAUNCHER__ makes Homebrew installs fragile

[Ronald Oussoren]

Ronald Oussoren  added the comment:

I had the some problem when using mypy. What does reproduce the issue is the 
following:

* Create *two* venv environments
* Install mypy in one of them
* Activate the other
* Run the subprocess call in the activated venv

I then get a message about not being able to find mypy:

.../some-env/bin/python3: No module named mypy.dmypy

I don't get the message when I call subprocess outside of a venv.

I haven't checked yet if my patch fixes this issue, that will have to wait for 
later this week.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22490] Using realpath for __PYVENV_LAUNCHER__ makes Homebrew installs fragile

[Guido van Rossum]

Guido van Rossum  added the comment:

I would love to know how to repro the original bug so I can demonstrate this 
actually fixes it.

Here's what I think should be the repro:

/Library/Frameworks/Python.framework/Versions/3.8/bin/python3  -c 'import 
subprocess; p = 
subprocess.Popen([".mypy/venv/bin/python3","-m","mypy.dmypy","-h"]).communicate()'

The setup is that .mypy/venv is a virtualenv that has mypy.dmypy installed. 
This is what failed for the user who reported this.

Unfortunately this does not fail for me (with a framework build from master 
installed). Would I have to do the install via brew? (I would have to learn 
about locally testing modified brew recipes, I suppose.)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22490] Using realpath for __PYVENV_LAUNCHER__ makes Homebrew installs fragile

[Ronald Oussoren]

Ronald Oussoren  added the comment:

I've added back port labels for 3.6 and 3.7. 

AFAIK 3.5 is closed for bugfixes at this point (except for security fixes).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22490] Using realpath for __PYVENV_LAUNCHER__ makes Homebrew installs fragile

[Guido van Rossum]

Guido van Rossum  added the comment:

Jason if you could test this we would be grateful!

Ronald what do you think of marking this as backportable to 3.7, 3.6 and
3.5?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22490] Using realpath for __PYVENV_LAUNCHER__ makes Homebrew installs fragile

[Ronald Oussoren]

Ronald Oussoren  added the comment:

To use the framework build you either have to install, use or set 
"DYLD_FRAMEWORK_PATH" in the environment (See the definition of RUNFORSHARED in 
Makefile for the value to use).

To properly test the this issue you have to install (or trick the system into 
thinking there is an install: I created a symlink in 
/Library/Frameworks/Python.framework/Versions: 
/Library/Frameworks/Python.framework/Versions/3.8 -> 
/Users/ronald/Projects/python/github/cpython-ronald/build/Python.framework/Versions/3.8).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22490] Using realpath for __PYVENV_LAUNCHER__ makes Homebrew installs fragile

[Jason R. Coombs]

Jason R. Coombs  added the comment:

Is it documented anywhere how to do a framework build of Python? When I try to 
do a framework build by running `./configure --enable-framework` then `make`, 
`./python.exe` emits the following:

dyld: Library not loaded: 
/Library/Frameworks/Python.framework/Versions/3.8/Python
  Referenced from: /Users/jaraco/code/public/cpython/./python.exe
  Reason: image not found
Aborted (core dumped)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22490] Using realpath for __PYVENV_LAUNCHER__ makes Homebrew installs fragile

[Ronald Oussoren]

Ronald Oussoren  added the comment:

I've added PR 9516 that just clears __PYVENV_LAUNCHER__ from the environment as 
soon as it is no longer needed.

A more involved change would be to change both the interpreter and the stub 
executable to avoid the need to use an environment variable in the first place:

- Add and "-X" flag to the interpreter to pass the information that's currently 
passed using an environment variable
- Change pythonw.c to add this new option to the argv vector.

This would be slightly cleaner, at the cost of having more complicated code 
(and is a change that would IMHO not qualify for a back port, while my current 
PR is minimal enough for a back port)

Note that both with and without this PR sys.executable points to the python 
executable inside the venv (when using a env).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22490] Using realpath for __PYVENV_LAUNCHER__ makes Homebrew installs fragile

[Ronald Oussoren]

Change by Ronald Oussoren :


--
pull_requests: +8922

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22490] Using realpath for __PYVENV_LAUNCHER__ makes Homebrew installs fragile

[Ronald Oussoren]

Ronald Oussoren  added the comment:

The pull request is wrong, the use of __PYVENV_LAUNCHER__ in pythonw.c is 
correct and should not be removed. Where the current code goes wrong is that it 
doesn't clear the environment as soon as possible.

This patch should basically do the trick (but requires testing and probably 
adjustments to some other code):

diff --git a/Modules/main.c b/Modules/main.c
index 3a6cf31fc8..c673d06c56 100644
--- a/Modules/main.c
+++ b/Modules/main.c
@@ -1360,6 +1360,8 @@ config_init_program_name(_PyCoreConfig *config)
  "variable", (Py_ssize_t)len);
 }
 config->program_name = program_name;
+
+   unsetenv("__PYVENV_LAUNCHER__");
 return _Py_INIT_OK();
 }
 }


Some background information on why and environment variable is used on macOS 
when using a framework install:  To use system GUI frameworks on macOS the 
executable should be part of an application bundle (".app"), otherwise a number 
of APIs just don't work (which would affect the use of for example Tkinter in 
command-line scripts).  To work around this platform limitation a framework 
install of Python has the actual interpreter in a Python.app bundle stored in 
the framework, and "{sys.prefix}/bin/python" is a small stub executable that 
launches the real interpreter. 

The environment variable is used to pass the path to the stub executable to the 
real interpreter, to enable it to behave correctly (look at venv configuration 
files, have sys.executable be correct, ...). 

That said, looking at the code it might be possible to do away with the 
environment variable after call because the code in pythonw.c suggests the the 
environment variable is only necessary for OSX 10.5 or earlier.

I'm working on an alternative pull request that implements the patch I included 
inline in this message, with some additional documentation.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22490] Using realpath for __PYVENV_LAUNCHER__ makes Homebrew installs fragile

[Jason R. Coombs]

Jason R. Coombs  added the comment:

I went naively through the codebase and removed every reference to 
PYVENV_LAUNCHER and submitted as PR 9498. Does the CI runner at 
https://python.visualstudio.com/cpython/_build/results?buildId=31019=logs 
use a framework build such that it's an adequate test of the change?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22490] Using realpath for __PYVENV_LAUNCHER__ makes Homebrew installs fragile

[Jason R. Coombs]

Change by Jason R. Coombs :


--
pull_requests: +8905
stage:  -> patch review

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22490] Using realpath for __PYVENV_LAUNCHER__ makes Homebrew installs fragile

[Vinay Sajip]

Vinay Sajip  added the comment:

As I said in msg284106, it seems that the __PYVENV_LAUNCHER__ should be removed 
from the stub launcher and then tests should confirm that framework builds 
aren't adversely affected. Unfortunately, I can't do this testing as I don't 
have a Mac, and have been hoping that one of the Mac platform experts (or 
indeed anyone who can build and test framework builds) can look at this.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22490] Using realpath for __PYVENV_LAUNCHER__ makes Homebrew installs fragile

[Guido van Rossum]

Guido van Rossum  added the comment:

I was pointed here after we found some erroneous behavior.

We have a script run by the system python that invokes a specific venv's 
python3 with a -m flag requesting a module that is installed (only) in that 
venv. A user complained that this failed, with the venv's python3 claiming the 
module was not installed (but it was, as proved by manually running it with the 
same -m flag).

Eventually someone realized that this was because the system python was a 
python3 installed by homebrew -- somehow this caused the system python (being 
python3) set __PYVENV_LAUNCHER__, and that made the venv's python3 ignore the 
venv's site-packages and instead look in the system python's site-packages.

This feels very wrong.

Maybe this is just a clearer description of https://bugs.python.org/issue31363? 
But that gained no traction while here there is at least some discussion.

--
nosy: +gvanrossum

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22490] Using realpath for __PYVENV_LAUNCHER__ makes Homebrew installs fragile

[Anders Hovmöller]

Change by Anders Hovmöller :


--
nosy: +Anders.Hovmöller

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22490] Using realpath for __PYVENV_LAUNCHER__ makes Homebrew installs fragile

[Anders Hovmöller]

Change by Anders Hovmöller :


--
versions: +Python 3.6

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22490] Using realpath for __PYVENV_LAUNCHER__ makes Homebrew installs fragile

[Vinay Sajip]

Vinay Sajip added the comment:

> I tried making a Framework build of Python with the environment variable 
> removed entirely, but I'm not experienced enough with builds on a Mac to know 
> what I need to do to create a proper build.

Given Jason's comment above, it's probably best for one of the Mac-expert 
committers to remove the environment variable altogether from 
Mac/Tools/pythonw.c:main(), and confirm everything still builds and runs OK, 
and then for Tim to confirm that Homebrew isn't adversely impacted by the 
change.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22490] Using realpath for __PYVENV_LAUNCHER__ makes Homebrew installs fragile

[Jason R. Coombs]

Jason R. Coombs added the comment:

That also looks good.

$ head -n1 ~/.envs/issue22490/bin/cherryd
#!/Users/jaraco/.envs/issue22490/bin/python

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22490] Using realpath for __PYVENV_LAUNCHER__ makes Homebrew installs fragile

[Vinay Sajip]

Vinay Sajip added the comment:

Thanks. It's worth checking what shebang was written to a script that was 
installed into the venv - e.g. cherryd.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22490] Using realpath for __PYVENV_LAUNCHER__ makes Homebrew installs fragile

[Jason R. Coombs]

Jason R. Coombs added the comment:

On Python 3.6, I made the edit. I actually used the one-liner `   
os.environ.pop('__PYVENV_LAUNCHER__', None)`. I then created a venv and 
installed a package and successfully ran a module in that package:

$ python -m venv ~/.envs/issue22490-test
$ ~/.envs/issue22490-test/bin/python -m pip install cherrypy
Collecting cherrypy
  Downloading CherryPy-8.5.0-py2.py3-none-any.whl (463kB)
100% || 471kB 2.2MB/s 
Collecting six (from cherrypy)
  Using cached six-1.10.0-py2.py3-none-any.whl
Installing collected packages: six, cherrypy
Successfully installed cherrypy-8.5.0 six-1.10.0
$ ~/.envs/issue22490-test/bin/python -m cherrypy
...
[27/Dec/2016:11:08:52] ENGINE Serving on http://127.0.0.1:8080
[27/Dec/2016:11:08:52] ENGINE Bus STARTED

So pyvenv seems to be working.

Additionally, it's corrected the undesirable behavior that Tim demonstrated:

$ rm -R ~/.envs/issue22490-test 
$ python -m virtualenv ~/.envs/issue22490-test
Using base prefix '/Library/Frameworks/Python.framework/Versions/3.6'
New python executable in /Users/jaraco/.envs/issue22490-test/bin/python3
Also creating executable in /Users/jaraco/.envs/issue22490-test/bin/python
Installing setuptools, pip, wheel...done.
$ python -c 'import subprocess; 
subprocess.call(["/Users/jaraco/.envs/issue22490-test/bin/python", "-c", 
"import sys; print(sys.executable)"])'
/Users/jaraco/.envs/issue22490-test/bin/python


I tried making a Framework build of Python with the environment variable 
removed entirely, but I'm not experienced enough with builds on a Mac to know 
what I need to do to create a proper build. I build Python from source and 
tried to create a venv, but it crashed with a nondescript error:

$ ./python.exe -VV
Python 3.6.0+ (3.6:86a412584c02+, Dec 27 2016, 11:24:01) 
[GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.42.1)]
$ ./python.exe -m venv ~/.envs/issue22490
Error: Command '['/Users/jaraco/.envs/issue22490/bin/python.exe', '-Im', 
'ensurepip', '--upgrade', '--default-pip']' returned non-zero exit status 1.

Aah. The issue is probably no zlib.

$ curl https://bootstrap.pypa.io/get-pip.py -o - | ~/.envs/issue22490/bin/python
  % Total% Received % Xferd  Average Speed   TimeTime Time  Current
 Dload  Upload   Total   SpentLeft  Speed
100 1558k  100 1558k0 0  5221k  0 --:--:-- --:--:-- --:--:-- 5210k
Traceback (most recent call last):
  File "", line 20061, in 
  File "", line 194, in main
  File "", line 82, in bootstrap
zipimport.ZipImportError: can't decompress data; zlib not available


After making sure I had zlib installed and available, by re-running 
xcode-select --install, I was able to build with zlib, create a venv, and 
install packages to it:

$ ./python.exe -m venv ~/.envs/issue22490 
$ ~/.envs/issue22490/bin/python -c "import os; 
os.environ['__PYVENV_LAUNCHER__']"
Traceback (most recent call last):
  File "", line 1, in 
  File "/Users/jaraco/Dropbox/code/public/cpython/Lib/os.py", line 669, in 
__getitem__
raise KeyError(key) from None
KeyError: '__PYVENV_LAUNCHER__'
$ ~/.envs/issue22490/bin/pip install cherrypy
Collecting cherrypy
  Using cached CherryPy-8.5.0-py2.py3-none-any.whl
Collecting six (from cherrypy)
  Using cached six-1.10.0-py2.py3-none-any.whl
Installing collected packages: six, cherrypy
Successfully installed cherrypy-8.5.0 six-1.10.0
$ ~/.envs/issue22490/bin/python -m cherrypy
...
[27/Dec/2016:11:58:09] ENGINE Serving on http://127.0.0.1:8080
[27/Dec/2016:11:58:09] ENGINE Bus STARTED


What else would I need to test? What might one expect to fail due to the lack 
of that environment variable?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22490] Using realpath for __PYVENV_LAUNCHER__ makes Homebrew installs fragile

[Vinay Sajip]

Vinay Sajip added the comment:

I'm not sure if msg230947 is still correct, and it seems neater to remove 
__PYVENV_LAUNCHER__ from where it's defined in the first place (the stub 
launcher C file) if it was just a shim needed around the time the functionality 
was developed (pre the 3.3 release).

It would be helpful if a Mac person could try removing it and seeing if the 
resulting framework builds of Python on OS X are adversely affected.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22490] Using realpath for __PYVENV_LAUNCHER__ makes Homebrew installs fragile

[Tim Smith]

Tim Smith added the comment:

Since __PYVENV_LAUNCHER__ is consulted in site.py, it seems likely that the 
latest it can be deleted is in site.py. The attached patch does that.

--
Added file: http://bugs.python.org/file46004/delete-venev-launcher.diff

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22490] Using realpath for __PYVENV_LAUNCHER__ makes Homebrew installs fragile

[Tim Smith]

Tim Smith added the comment:

I spoke prematurely; I recently rediscovered that the persistence of 
__PYVENV_LAUNCHER__ poisons the sys.executable of virtualenv interpreters 
launched as a subprocess of another Python interpreter:

$ virtualenv -p python3 test
$ test/bin/python3 -c 'import sys; print(sys.executable)'
/Users/tim/test/bin/python3

$ /usr/local/bin/python3 -c 'import subprocess; 
subprocess.call(["/Users/tim/test/bin/python3", "-c", "import sys; 
print(sys.executable)"])'
/usr/local/bin/python3

$ /usr/local/bin/python3 -c 'import subprocess, os; del 
os.environ["__PYVENV_LAUNCHER__"]; 
subprocess.call(["/Users/tim/test/bin/python3", "-c", "import sys; 
print(sys.executable)"])'
/Users/tim/test/bin/python3

If __PYVENV_LAUNCHER__ can be unset before script execution begins, that seems 
ideal.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22490] Using realpath for __PYVENV_LAUNCHER__ makes Homebrew installs fragile

[Jason R. Coombs]

Jason R. Coombs added the comment:

I believe this behavior (presence of the __PYVENV_LAUNCHER__ and inheritance in 
child processes) continues to cause problems. See 
https://github.com/pypa/virtualenv/issues/845 for a simple reproducing of the 
issue, even on pip 6+.

--
nosy: +jason.coombs

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22490] Using realpath for __PYVENV_LAUNCHER__ makes Homebrew installs fragile

[Tim Smith]

Tim Smith added the comment:

Homebrew's interest in this ticket was resolved by the release of pip 6, which 
includes Vinay's change to distlib to use sys.executable instead of 
__PYVENV_LAUNCHER__. Many thanks!

I'm not marking this fixed in case it is useful to leave this open to discuss 
unsetting __PYVENV_LAUNCHER__.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22490
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22490] Using realpath for __PYVENV_LAUNCHER__ makes Homebrew installs fragile

[Ronald Oussoren]

Ronald Oussoren added the comment:

The environment variable itself cannot be removed from CPython, it is necessary 
to implement the correct behavior of pyvenv with framework builds of Python.

That said, I do think that the environment variable should be unset as soon as 
possible in the CPython startup code to avoid accidentally affecting other 
interpreters.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22490
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22490] Using realpath for __PYVENV_LAUNCHER__ makes Homebrew installs fragile

[Vinay Sajip]

Vinay Sajip added the comment:

 I don't quite understand why you use __PYVENV_LAUNCHER__

When I first developed the venv functionality it was definitely needed, but it 
looks as if it is not needed now. Perhaps to fix this completely, the following 
needs to be done, on the assumption that __PYVENV_LAUNCHER__ is no longer 
needed:

1. Remove the reference to it in distlib, and use sys.executable instead. Once 
pip incorporates this fix, the Homebrew problem should go away. (I have already 
made the change in the distlib repo, but this needs to be released in order for 
pip to consider vendoring it, and then pip needs to be released before Python 
can incorporate it).

2. Remove references to the environment variable in Python itself, using 
sys.executable instead.

As the env var was an implementation detail, ISTM it could be removed in a 3.4 
point release - would you agree?

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22490
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22490] Using realpath for __PYVENV_LAUNCHER__ makes Homebrew installs fragile

[Ronald Oussoren]

Ronald Oussoren added the comment:

First of all, sorry about the slow response.

Vinay: I don't quite understand why you use __PYVENV_LAUNCHER__: When I create 
a pyvenv using python 3.3 (with a oldish build from the 3.3 branch) 
sys.executable point to a binary in the venv without mucking with environment 
variables:

$ pyvenv-3.3 testenv
$ testenv/bin/python
Python 3.3.5+ (3.3:a36d469f31c1, Aug 13 2014, 09:04:41) 
[GCC 4.2.1 Compatible Apple LLVM 5.1 (clang-503.0.40)] on darwin
Type help, copyright, credits or license for more information.
 import sys
 sys.executable
'/Users/ronald/Projects/pyobjc-hg/pyobjc/x/testenv/bin/python'


If older version of Python 3.x don't do this that's a bug in those versions and 
it would be acceptable to use the environment variable with those versions to 
work around this bug.

I still not convinced that __PYVENV_LAUNCHER__ needs to be changed. It's highly 
likely that any use of __PYVENV_LAUNCHER__ outside of CPython itself is an 
indication of a bug in CPython that needs to be fixed there.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22490
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22490] Using realpath for __PYVENV_LAUNCHER__ makes Homebrew installs fragile

[Ned Deily]

Ned Deily added the comment:

A comment on Vinay's comment: and this cannot be obtained from sys.executable 
because that is pointing to a framework executable.  That *was* true prior to 
3.3 and is still true at the moment for 2.7.x but it is not true for 3.3+.  
Ronald's change (b79d276041a8) in Issue15307 for venv framework support changed 
the stub launcher to no longer do a realpath() on the executable name which 
means that sys.executable contains the path to the Python stub launcher whether 
or not in a venv:

$ ls -l /usr/local/bin/python2.7
lrwxr-xr-x  1 root  admin  71 Jul  3 01:21 /usr/local/bin/python2.7 - 
../../../Library/Frameworks/Python.framework/Versions/2.7/bin/python2.7
$ /usr/local/bin/python2.7 -c 'import sys;print(sys.executable)'
/Library/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python

$ ls -l /usr/local/bin/python3.4
lrwxr-xr-x  1 root  wheel  71 Oct  6 16:53 /usr/local/bin/python3.4 - 
../../../Library/Frameworks/Python.framework/Versions/3.4/bin/python3.4
$ /usr/local/bin/python3.4 -c 'import sys;print(sys.executable)'
/usr/local/bin/python3.4

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22490
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22490] Using realpath for __PYVENV_LAUNCHER__ makes Homebrew installs fragile

[Vinay Sajip]

Vinay Sajip added the comment:

 Ronald's change ... changed the stub launcher to no longer do a realpath() on 
 the executable name
 which means that sys.executable contains the path to the Python stub launcher 
 whether or not in a venv

Ronald's change doesn't do a realpath() on the executable name itself, but it 
*does* do a realpath() on the executable's directory. This seems to be what's 
causing the problem in Homebrew builds: there are two levels of symlink at play.

/usr/local/bin/python3.4 - /usr/local/opt/python3/bin/python3.4 - 
/usr/local/Cellar/python3/3.4.1_1/Frameworks/Python.framework/Versions/3.4/bin/python3.4

Because of the realpath(), the wrong stub launcher (from Homebrew's 
perspective) gets pointed to by the environment variable. At least, if my 
understanding is correct.

https://github.com/pypa/pip/issues/2031#issuecomment-56294599

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22490
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22490] Using realpath for __PYVENV_LAUNCHER__ makes Homebrew installs fragile

[Tim Smith]

Tim Smith added the comment:

We would like to refer to python3 as /usr/local/opt/python3/bin/python3, where 
/usr/local/opt/python3 is a symlink to ../Cellar/python3/3.4.2, and 
/usr/local/Cellar/python3/3.4.2/bin/python3 is a symlink to 
/usr/local/Cellar/python3/3.4.2/Frameworks/Python.framework/Versions/3.4/bin/python3,
 which is the stub launcher for 
/usr/local/Cellar/python3/3.4.2/Frameworks/Python.framework/Versions/3.4/Python.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22490
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22490] Using realpath for __PYVENV_LAUNCHER__ makes Homebrew installs fragile

[Tim Smith]

Tim Smith added the comment:

I'm attaching an updated patch; it passes tests for me locally with a framework 
build.

--
Added file: http://bugs.python.org/file36885/dont-realpath-venv-dirname.diff-1

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22490
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22490] Using realpath for __PYVENV_LAUNCHER__ makes Homebrew installs fragile

[Tim Smith]

Tim Smith added the comment:

Er, because the test has been modified by taking Vinay's suggestion to test 
that the directories are physically identical instead of doing a string 
comparison.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22490
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22490] Using realpath for __PYVENV_LAUNCHER__ makes Homebrew installs fragile

[Ned Deily]

Ned Deily added the comment:

From the initial description of the problem, it's not clear to me that there 
is a problem here needing resolution in the stub launcher.  I've asked for 
clarification on the pip issue tracker.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22490
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22490] Using realpath for __PYVENV_LAUNCHER__ makes Homebrew installs fragile

[Ned Deily]

Ned Deily added the comment:

Also, the patch causes a test failure with a framework build:

==
FAIL: test_defaults (test.test_venv.BasicTest)
--
Traceback (most recent call last):
  File 
/py/dev/3x/root/fwd/Library/Frameworks/pytest_10.9.framework/Versions/3.5/lib/python3.5/test/test_venv.py,
 line 106, in test_defaults
self.assertIn('home = %s' % path, data)
AssertionError: 'home = /py/dev/3x/root/fwd/./bin' not found in 'home = 
/py/dev/3x/root/fwd/bin\ninclude-system-site-packages = false\nversion = 
3.5.0\n'

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22490
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22490] Using realpath for __PYVENV_LAUNCHER__ makes Homebrew installs fragile

[Ronald Oussoren]

Ronald Oussoren added the comment:

I'm not convinced that this is a bug in python. __PYVENV_LAUNCHER__ is an 
implementation detail of CPython used in the implementation of the pyvenv 
functionality. 

I consider using this undocumented environment variable in distlib as a bug in 
distlib, which should be fixed there.  

It would be nice to know why distlib uses __PYVENV_LAUNCHER__ as that could 
lead us to a real bug or missing feature. 

BTW. I haven't read the discussion in [2] yet.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22490
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22490] Using realpath for __PYVENV_LAUNCHER__ makes Homebrew installs fragile

[Ronald Oussoren]

Ronald Oussoren added the comment:

 On 25 sep. 2014, at 19:58, Ronald Oussoren rep...@bugs.python.org wrote:
 
 
 Ronald Oussoren added the comment:
 
 I consider using this undocumented environment variable in distlib as a bug 
 in distlib, which should be fixed 

Speaking of which... That environment variable shouldn't leak into Python code 
in the first place, launching a regular Python interpreter from a venv could 
currently result in unwanted behavior.  I'll try to provide an example of that. 

Ronald
 
 --
 
 ___
 Python tracker rep...@bugs.python.org
 http://bugs.python.org/issue22490
 ___

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22490
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22490] Using realpath for __PYVENV_LAUNCHER__ makes Homebrew installs fragile

[Vinay Sajip]

Vinay Sajip added the comment:

 I consider using this undocumented environment variable in distlib
 as a bug in distlib, which should be fixed there.  

Well, let me explain why it's used: when Python = 3.3 starts up, it looks for 
a pyvenv.cfg file proximate to sys.executable. If found, that means we're in a 
venv, and sys.path is set up accordingly. On OS X, the stub launcher is 
copied/symlinked to the venv when a new venv is created - the real interpreter 
is not copied. So, shebangs written by distlib into scripts installed into a 
venv must be of the form #!/path/to/venv/bin/python, and this cannot be 
obtained from sys.executable because that is pointing to a framework 
executable. There would be no pyvenv.cfg anywhere near that location.

This is why the __PYVENV_LAUNCHER__ variable was created, and distlib uses it 
because it needs to conform to how venvs work. In this respect distlib is a bit 
like setuptools - it needs to understand some low-level details which other 
libraries don't need to worry about.

Scripts installed in venvs work as expected (AFAICT) when used with stock 
Python framework builds on OS X. With HomeBrew, the complication appears to be 
caused by two levels of symlink: the executable in /usr/local/ points to one in 
/usr/local/Cellar/..., which in turn points to the framework executable.

The failing test (test_defaults) could be fixed by looking for equivalence in 
the home = directories in the test, rather than a string-contains-value test 
as at present.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22490
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22490] Using realpath for __PYVENV_LAUNCHER__ makes Homebrew installs fragile

[Tim Smith]

New submission from Tim Smith:

Homebrew, the OS X package manager, distributes python3 as a framework build. 
We like to be able to control the shebang that gets written to scripts 
installed with pip. [1]

The path we prefer for invoking the python3 interpreter is like 
/usr/local/opt/python3/bin/python3.4, which is symlinked to the framework stub 
launcher at 
/usr/local/Cellar/python3/3.4.1_1/Frameworks/Python.framework/Versions/3.4/bin/python3.4.
 For Python 2.x, we discovered that assigning 
/usr/local/opt/python/bin/python2.7 to sys.executable in sitecustomize.py 
resulted in correct shebangs for scripts installed by pip. The same approach 
doesn't work with Python 3.

A very helpful conversation with Vinay Sajip [2] led us to consider how the 
python3 stub launcher sets __PYVENV_LAUNCHER__, which distlib uses in 
preference to sys.executable to discover the intended interpreter when pip 
writes shebangs.

Roughly, __PYVENV_LAUNCHER__ is set from argv[0], so it mimics the invocation 
of the stub, except that symlinks in the directory component of the path to the 
identified interpreter are resolved to a real path. For us, this means that 
__PYVENV_LAUNCHER__ (and therefore the shebangs of installed scripts) always 
points to the Cellar path, not the preferred opt path, even when python is 
invoked via the opt path.

Avoiding this symlink resolution would allow us to control pip's shebang (which 
sets the shebangs of all pip-installed scripts) by controlling the way we 
invoke python3 when we use ensurepip during installation.

Building python3 with the attached diff removes the symlink resolution.

[1]  This is important to Homebrew because packages are physically installed to 
versioned prefixes, like /usr/local/Cellar/python3/3.4.1_1/. References to 
these real paths are fragile and break when the version number changes or when 
the revision number (_1) changes, when can happen when e.g. openssl is 
released and Python needs to be recompiled against the new library. To avoid 
this breakage, Homebrew maintains a version-independent symlink to each 
package, like /usr/local/opt/python3, which points to the 
.../Cellar/python3/3.4.1_1/ location.

[2] https://github.com/pypa/pip/issues/2031

--
assignee: ronaldoussoren
components: Macintosh
files: dont-realpath-venv-dirname.diff
keywords: patch
messages: 227505
nosy: ned.deily, ronaldoussoren, tdsmith, vinay.sajip
priority: normal
severity: normal
status: open
title: Using realpath for __PYVENV_LAUNCHER__ makes Homebrew installs fragile
type: behavior
versions: Python 3.4, Python 3.5
Added file: http://bugs.python.org/file36718/dont-realpath-venv-dirname.diff

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22490
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com