[issue42013] venv on Windows with symlinks is broken if invoked with -I

[Eryk Sun]

Eryk Sun  added the comment:

> So I'm +1 on fixing this by calling realpath.

In POSIX, calculate_path() in Modules/getpath.c calls calculate_argv0_path() 
before it calls calculate_read_pyenv(), and calculate_argv0_path() in turn 
calls resolve_symlinks(>argv0_path). Thus "pyvenv.cfg" isn't found 
and isn't used to find the standard library in POSIX when a virtual environment 
uses symlinks. Later on, "pyvenv.cfg" gets read by the site module, if 
importing the latter isn't skipped via -S. The site module sets sys._home to 
the `home` value, but sys._home only appears to be used by sysconfig when 
running from the build directory.

In Windows, calculate_path() in PC/getpathp.c could get the final (real) path 
for argv0_path before calling calculate_pyvenv_file(). Then, just like in 
POSIX, the environment's "pyvenv.cfg" file won't be found and won't be used to 
find the standard library when a virtual environment uses symlinks.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42013] venv on Windows with symlinks is broken if invoked with -I

[Paul Moore]

Paul Moore  added the comment:

I'm inclined to think that creating a venv from within another venv should be 
allowed. Tools like pipx can result in *other* tools being run from within a 
virtual environment, and I don't think it's good to disallow that usage - as an 
example, I have virtualenv, tox and nox installed via pipx, and all of them 
create virtual environments.

So I'm +1 on fixing this by calling realpath.

I don't think we need to do anything special for --system-site-packages. The 
docs say "Give the virtual environment access to the system site-packages dir". 
If people are trying to chain venvs using it, they are misreading that comment 
- and the best they could hope for is to request that it's made clearer.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42013] venv on Windows with symlinks is broken if invoked with -I

[Steve Dower]

Steve Dower  added the comment:

Thanks for figuring that out, Eryk.

Probably we should just update venv to do a realpath(sys._base_executable) to 
handle the venv-from-symlinked-venv scenario.

Though I'd also be quite happy to just disallow that entirely (as we used to?). 
If you enable system site packages thinking you're chaining venvs together, you 
may be surprised, so it might just be better to error out here. (On the *other* 
hand, if you're programmatically invoking venv, maybe you know what you're 
doing and we should allow it?)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42013] venv on Windows with symlinks is broken if invoked with -I

[Eryk Sun]

Eryk Sun  added the comment:

This issue is partly due to bpo-8901, which changed the behavior of the -E and 
-I command-line options to make them ignore the default PythonPath value in the 
registry key "Software\Python\PythonCore\X.Y\PythonPath". The change itself is 
not wrong. It's just exposing an underlying problem.

The `home` path in pyvenv.cfg is from sys._base_executable. In a launcher-based 
environment that's created from the base installation, sys._base_executable is 
the real base executable. OTOH, in a symlink-based virtual environment, 
sys._base_executable is the same as sys.executable. Consequently, if a virtual 
environment is created from a symlink-based virtual environment, the `home` 
path in pyvenv.cfg refers to the creating environment instead of the base 
installation. In this case, with the current implementation, the standard 
library can only be found by falling back on the default PythonPath in the 
registry.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42013] venv on Windows with symlinks is broken if invoked with -I

[Eryk Sun]

Change by Eryk Sun :


--
Removed message: https://bugs.python.org/msg378489

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42013] venv on Windows with symlinks is broken if invoked with -I

[Eryk Sun]

Eryk Sun  added the comment:

> Well, actually the environment variables /should/ not matter as -I 
> implies -E.

The operative word there is "should". I was grasping for anything that might 
explain why I couldn't reproduce the issue.

> Ok, the missing link is that the python you run into needs to 
> be also a symlink venv

Thank you, that reproduces the problem for me.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42013] venv on Windows with symlinks is broken if invoked with -I

[Bernat Gabor]

Bernat Gabor  added the comment:

> We need more information about the specific environment this is triggering.

Both Gitub Actions Windows CPython3.9 or installer as downloaded from (on 
Windows 10) https://www.python.org/downloads/release/python-390/.

> PYTHONHOME should not be set all. PYTHONPATH needs to be set carefully. It 
> should never include standard-library path

The issue manifests independent of those environment variables, netiher of them 
are set.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42013] venv on Windows with symlinks is broken if invoked with -I

[Bernat Gabor]

Bernat Gabor  added the comment:

Ok, the missing link is that the python you run into needs to be also a symlink 
venv:

❯ py -m venv env --without-pip --clear --symlinks
❯ .\env\Scripts\python.exe -c "import venv, subprocess; 
venv.EnvBuilder(with_pip=False, symlinks=True).create('venv'); 
subprocess.check_call(['venv\\Scripts\\python.exe', '-Ic', 'import sys; 
print(sys.executable)'])"

Fatal Python error: init_fs_encoding: failed to get the Python codec of the 
filesystem encoding
Python runtime state: core initialized
ModuleNotFoundError: No module named 'encodings'

--
nosy:  -FFY00

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42013] venv on Windows with symlinks is broken if invoked with -I

[Filipe Laíns]

Filipe Laíns  added the comment:

Well, actually the environment variables /should/ not matter as -I implies -E.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42013] venv on Windows with symlinks is broken if invoked with -I

[Filipe Laíns]

Filipe Laíns  added the comment:

This error most likely happens because sys.path isn't being set properly and 
the standard library isn't being included.

We need more information about the specific environment this is triggering.

Which CPython build are you using? The one from Github Actions, right?
We also need the environment variables.

--
nosy: +FFY00

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42013] venv on Windows with symlinks is broken if invoked with -I

[Eryk Sun]

Eryk Sun  added the comment:

What about the PYTHONHOME and PYTHONPATH environment variables?

>>> 'PYTHONHOME' in os.environ
False
>>> 'PYTHONPATH' in os.environ
False

PYTHONHOME should not be set all. PYTHONPATH needs to be set carefully. It 
should never include standard-library paths.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42013] venv on Windows with symlinks is broken if invoked with -I

[Bernat Gabor]

Bernat Gabor  added the comment:

❯ py -c 'import sys; print(sys.version)'
3.9.0 (tags/v3.9.0:9cf6752, Oct  5 2020, 15:34:40) [MSC v.1927 64 bit (AMD64)]

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42013] venv on Windows with symlinks is broken if invoked with -I

[Eryk Sun]

Eryk Sun  added the comment:

I can't reproduce the issue with the normal 3.9.0 distribution from python.org. 
For example:

>>> venv.EnvBuilder(with_pip=False, symlinks=True).create("venv")
>>> subprocess.check_call(["venv\\Scripts\\python.exe", "-Ic", "import sys; 
print(sys.executable)"])
C:\Temp\env\venv\Scripts\python.exe
0

Which Python version(s) and distribution(s) did you test? Do you have the 
PYTHONHOME and/or PYTHONPATH environment variables set?

--
nosy: +eryksun

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42013] venv on Windows with symlinks is broken if invoked with -I

[Bernat Gabor]

Change by Bernat Gabor :


--
components: +Windows
nosy: +paul.moore, steve.dower, tim.golden, zach.ware

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42013] venv on Windows with symlinks is broken if invoked with -I

[Bernat Gabor]

New submission from Bernat Gabor :

Here's a small reproducible, run it on a Windows OS that has symlinks enabled:

import shutil
import venv
import subprocess

shutil.rmtree("venv", ignore_errors=True)
venv.EnvBuilder(with_pip=False, symlinks=True).create("venv")

# works
subprocess.check_call(["venv\\Scripts\\python.exe", "-c", "import sys; 
print(sys.executable)"])

# fails with No module named 'encodings'
subprocess.check_call(["venv\\Scripts\\python.exe", "-Ic", "import sys; 
print(sys.executable)"])

--
messages: 378485
nosy: gaborjbernat
priority: normal
severity: normal
status: open
title: venv on Windows with symlinks is broken if invoked with -I
versions: Python 3.10, Python 3.9

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com