[issue30811] A venv created and activated from within a virtualenv uses the outer virtualenv's site-packages rather than its own.

[Tzu-ping Chung]

Tzu-ping Chung  added the comment:

Unfortunately it is not a viable solution. If you run the virtualenv python 
without activation, e.g.

virtualenv --python=python3.7 foo
foo/bin/python -m venv bar

The VIRTUAL_ENV environment variable wouldn’t be set in this situation, but the 
created venv (bar) would still be broken.

The only viable cue I find to detect virtualenv existence is it sets 
sys.real_prefix to point to the prefix of the actual Python isntallation (i.e. 
the value of sys.prefix without virtualenv), while this variable does not exist 
by default. This is, however, obviously very hacky and unreliable, and not 
something venv should do IMO. virtualenv is the problem here, and should be 
responsible for fixing this situation instead.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30811] A venv created and activated from within a virtualenv uses the outer virtualenv's site-packages rather than its own.

[Ilya Kulakov]

Ilya Kulakov  added the comment:

Also hit this issue while trying to run venv on Travis.

Perhaps venv should detect if it's running under virtualenv (e.g. the 
VIRTUAL_ENV env is present) and issue a warning?

--
nosy: +Kentzo

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30811] A venv created and activated from within a virtualenv uses the outer virtualenv's site-packages rather than its own.

[Vinay Sajip]

Vinay Sajip  added the comment:

I'm closing this, as it's not a bug in the venv module.

--
resolution:  -> not a bug
stage:  -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30811] A venv created and activated from within a virtualenv uses the outer virtualenv's site-packages rather than its own.

[Josh Holland]

Change by Josh Holland :


--
nosy: +anowlcalledjosh

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30811] A venv created and activated from within a virtualenv uses the outer virtualenv's site-packages rather than its own.

[Tzu-ping Chung]

Tzu-ping Chung  added the comment:

@cosven you are correct! I made some additional observation while working on 
adding venv support to pew, in this thread: 
https://github.com/berdario/pew/pull/173. I’ll try to summarise below.

The root problem is indeed virtualenv’s custom site module, but even then it is 
possible for venv (or any tool built around it) to know where the “original” 
Python is, since virtualenv stores the old sys.prefix in sys.real_prefix. 
Unfortunately, however, this is not as useful as it seems. What we really want 
in this situation is sys.exec_prefix, which may or may not be the same as 
sys.prefix. I did manage to cobble together a hack, but it is by no means 
bullet-proof.

To actually deal with this problem, we will need to amend the site module 
provided by virtualenv, as you mentioned. But this still can’t “solve” it. 
Since the site module is vendored in the produced virtualenv, existing 
virtualenvs will stay broken even after we managed to upgrade the virtualenv 
installations.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30811] A venv created and activated from within a virtualenv uses the outer virtualenv's site-packages rather than its own.

[cosven]

cosven  added the comment:

It is actually a bug in virtualenv instead of venv.

> the outer environment's pip leaks into the inner environment;

This is true, but why? In short, the inner-env python binary
use `outer-env/` directory as its `prefix` direcotry,
but the `inner-env/` directory is the right `prefix` directory.

When the Python binary is executed, it attempts to determine
its `prefix` (which it stores in sys.prefix), which is then used
to find the standard library and other key files, and by the
`site` module to determine the location of the site-package directories.

However, virtualenv has its own `site` module, which is different
implemented from the site module in stdlib. It makes the inner-python
get a wrong `prefix` value.

> (Could a fix on virtualenv's side help?)
In my own opinion, if virtualenv change its implementation of site.py,
there is a change to fix this.

--
nosy: +cosven

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30811] A venv created and activated from within a virtualenv uses the outer virtualenv's site-packages rather than its own.

[Tzu-ping Chung]

Tzu-ping Chung  added the comment:

Plot twist: the workaround does not work on Windows.

$ cd SOMEDIR   ; Placeholder for a valid directory.
$ virtualenv env1
...
$ env\Scripts\python.exe
Python 3.6.4 (v3.6.4:d48eceb, Dec 19 2017, 06:54:40) [MSC v.1900 64 bit 
(AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import sysconfig
>>> print(sysconfig.get_config_var('BINDIR'))
SOMDIR\env1\Scripts

This points to the virtualenv's BINDIR, not the real Python installation's :(

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30811] A venv created and activated from within a virtualenv uses the outer virtualenv's site-packages rather than its own.

[Antony Lee]

Change by Antony Lee :


--
nosy:  -Antony.Lee

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30811] A venv created and activated from within a virtualenv uses the outer virtualenv's site-packages rather than its own.

[Tzu-ping Chung]

Tzu-ping Chung  added the comment:

Would it be possible for venv to do this automatically? Instead of using 
sys.executable, use

import sysconfig
dirname, exename = sysconfig.get_config_vars('BINDIR', 'BUILDPYTHON')

to populate values in the context?

--
nosy: +uranusjr

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30811] A venv created and activated from within a virtualenv uses the outer virtualenv's site-packages rather than its own.

[Vinay Sajip]

Vinay Sajip  added the comment:

> I guess it's reasonable, I'll see whether we can use the workaround you 
> proposed.

OK, please post your findings here.

> (Could a fix on virtualenv's side help?)

Probably not, as it's not behaving unexpectedly. It might be worth asking the 
Travis devs for an enhancement to allow the choice of -m venv rather than 
virtualenv to create venvs for Python >= 3.3, via an option in .travis.yml.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30811] A venv created and activated from within a virtualenv uses the outer virtualenv's site-packages rather than its own.

[Antony Lee]

Antony Lee  added the comment:

I guess it's reasonable, I'll see whether we can use the workaround you 
proposed.  (Could a fix on virtualenv's side help?)
Thanks for the explanations.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30811] A venv created and activated from within a virtualenv uses the outer virtualenv's site-packages rather than its own.

[Vinay Sajip]

Vinay Sajip  added the comment:

Is it OK if I close this? It's not really a bug, nor a case that's been 
designed for (a stdlib module supporting specific external packages is very 
unusual - pip support via ensurepip is perhaps the one exception I can think 
of). Given there's a way to create a venv from a virtualenv's Python, as posted 
above, I'm not sure what more I can do.

If, on the other hand, you look into the details and find something I've 
missed, I'll certainly look again.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30811] A venv created and activated from within a virtualenv uses the outer virtualenv's site-packages rather than its own.

[Vinay Sajip]

Vinay Sajip  added the comment:

> I don't understand why things would be different when nesting?

Specifically because venv keeps "a pointer" to the Python environment it was 
created from. Usually that's a system Python. If a venv ("inner") is created 
from a virtualenv's interpreter, the pointer points back to that environment 
("outer", in this case).

Also, virtualenv can re-invoke itself with a different interpreter easily - 
that's business as usual. However, the venv module is part of a specific Python 
stdlib and doesn't reinvoke itself with a different interpreter.

To find the exact mechanisms which lead to these behaviours, you would need to 
examine the code of virtualenv and the venv module and perhaps do some stepping 
through in a debugger!

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30811] A venv created and activated from within a virtualenv uses the outer virtualenv's site-packages rather than its own.

[Antony Lee]

Antony Lee  added the comment:

> venv, on the other hand, doesn't, but keeps a reference to its original 
> Python environment. That is, I think, the reason for the difference in 
> behaviour.

But for example, using the system Python to create a venv (no nesting), 
packages installed system-wide (e.g. using a linux package manager) are not 
visible in the venv.  Or, as you note, nesting a venv into another works fine.  
I don't understand why things would be different when nesting?

In any case thanks for the workaround.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30811] A venv created and activated from within a virtualenv uses the outer virtualenv's site-packages rather than its own.

[Vinay Sajip]

Vinay Sajip  added the comment:

The problem for venv is that it's tied to the running interpreter, which is (in 
the case you mention) the one from the outer virtualenv. Unlike virtualenv, 
venv does not provide a mechanism to restart itself with a different Python 
interpreter.

I don't know how Conda works in detail, but virtualenv makes copies of parts of 
the stdlib; venv, on the other hand, doesn't, but keeps a reference to its 
original Python environment. That is, I think, the reason for the difference in 
behaviour.

Instead of just using 'python' to invoke the command which runs the "-mvenv", 
you could get the underlying Python and create the venv using that. For 
example, using this script "upvenv.py":

#!/usr/bin/env python
def make_venv(venvpath):
import os
import subprocess
import sysconfig

python = os.path.join(sysconfig.get_config_var('BINDIR'), 'python3')
cmd = [python, '-mvenv', venvpath]
subprocess.run(cmd)

if __name__ == '__main__':
import sys
assert len(sys.argv) == 2
try:
rc = make_venv(sys.argv[1])
except Exception as e:
print('Failed: %s' % e)
rc = 1
sys.exit(rc)

I get:

vinay@ubuntu:/tmp$ cd /tmp
vinay@ubuntu:/tmp$ ve15 --version
15.1.0
vinay@ubuntu:/tmp$ ve15 -p python3 outer
Running virtualenv with interpreter /usr/bin/python3
Using base prefix '/usr'
New python executable in /tmp/outer/bin/python3
Also creating executable in /tmp/outer/bin/python
Installing setuptools, pip, wheel...done.
vinay@ubuntu:/tmp$ source outer/bin/activate
(outer)  vinay@ubuntu:/tmp$ python upvenv.py inner
(outer)  vinay@ubuntu:/tmp$ source inner/bin/activate
(inner)  vinay@ubuntu:/tmp$ python -minspect -d pip
Target: pip
Origin: /tmp/inner/lib/python3.5/site-packages/pip/__init__.py
Cached: 
/tmp/inner/lib/python3.5/site-packages/pip/__pycache__/__init__.cpython-35.pyc
Loader: <_frozen_importlib_external.SourceFileLoader object at 0x7f6917e9e860>
Submodule search path: ['/tmp/inner/lib/python3.5/site-packages/pip']

which seems OK.

This approach can't be transplanted into venv because it doesn't have the 
ability to restart with a different interpreter.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30811] A venv created and activated from within a virtualenv uses the outer virtualenv's site-packages rather than its own.

[Antony Lee]

Antony Lee  added the comment:

Travis uses virtualenvs as toplevel containers for running CI.  Some projects 
need (or would like to) set up venvs as part of their test suites.  On example 
is https://github.com/airspeed-velocity/asv, which profiles a project's speed 
across revisions; in order to do so it sets up an environment for each revision 
to run the test suite.  Currently, it can either set up a virtualenv or a conda 
env, but it would make sense for it to be able to set up a venv too 
(https://github.com/airspeed-velocity/asv/pull/526).  But the unability to nest 
a venv into a virtualenv makes this impossible to test on Travis.

--
status: pending -> open

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30811] A venv created and activated from within a virtualenv uses the outer virtualenv's site-packages rather than its own.

[Vinay Sajip]

Vinay Sajip  added the comment:

Can you explain why virtualenv and venv need to be mixed in this way, other 
than as an academic exercise? If you use -m venv for both inner and outer 
venvs, then it seems to work as expected.

I'm planning to close this as "not a bug" unless you can provide a reason why 
this use case needs to be supported.

--
status: open -> pending

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30811] A venv created and activated from within a virtualenv uses the outer virtualenv's site-packages rather than its own.

[Berker Peksag]

Change by Berker Peksag :


--
nosy: +vinay.sajip

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30811] A venv created and activated from within a virtualenv uses the outer virtualenv's site-packages rather than its own.

[Antony Lee]

Antony Lee added the comment:

Sorry, that was a sloppy report.  This is a better repro:

$ export PIP_CONFIG_FILE=/dev/null  # just to be sure
python -mvirtualenv outer-env  # using /usr/bin/python(3)
source outer-env/bin/activate
python -mvenv inner-env  # using outer-env's python
source inner-env/bin/activate
python -minspect -d pip

Using base prefix '/usr'
New python executable in /tmp/outer-env/bin/python
Installing setuptools, pip, wheel...done.
Target: pip
Origin: /tmp/outer-env/lib/python3.6/site-packages/pip/__init__.py
Cached: 
/tmp/outer-env/lib/python3.6/site-packages/pip/__pycache__/__init__.cpython-36.pyc
Loader: <_frozen_importlib_external.SourceFileLoader object at 0x7f92e00e03c8>
Submodule search path: ['/tmp/outer-env/lib/python3.6/site-packages/pip']

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30811] A venv created and activated from within a virtualenv uses the outer virtualenv's site-packages rather than its own.

[Antony Lee]

New submission from Antony Lee:

Python 3.6.1, virtualenv 15.1.0 (Arch Linux, distro packages)

```
export PIP_CONFIG_FILE=/dev/null  # just to be sure
# python -mvirtualenv outer-env  # using /usr/bin/python(3)
python2 -mvirtualenv outer-env  # using /usr/bin/python(3)
source outer-env/bin/activate
python -mvenv inner-env  # using outer-env's python
source inner-env/bin/activate
python -minspect -d pip
```

The last step outputs
```
Target: pip
Origin: /tmp/inner-env/lib/python3.6/site-packages/pip/__init__.py
Cached: 
/tmp/inner-env/lib/python3.6/site-packages/pip/__pycache__/__init__.cpython-36.pyc
Loader: <_frozen_importlib_external.SourceFileLoader object at 0x7f27c019c710>
Submodule search path: ['/tmp/inner-env/lib/python3.6/site-packages/pip']
```
i.e., the outer environment's pip leaks into the inner environment; the inner 
environment does not have its own pip installed.  Trying to use that pip from 
the inner environment (from inner-env, `python -mpip install ...`) results in 
the package being installed in the outer environment rather than the inner one.

Now this may all seem very academic, but for example Travis uses virtualenvs as 
toplevel containers for their Python projects, so this bug makes it difficult 
to test, say, projects that set up venvs as part of their workflow.

--
components: Library (Lib)
messages: 297365
nosy: Antony.Lee
priority: normal
severity: normal
status: open
title: A venv created and activated from within a virtualenv uses the outer 
virtualenv's site-packages rather than its own.
versions: Python 3.6

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com