[issue30628] why venv install old pip?

[Nick Coghlan]

Nick Coghlan added the comment:

Curtis: by design, ``ensurepip`` isn't permitted to assume internet access 
(since we want both it and venv to work without warnings or errors on systems 
with no internet access). This was covered in the original PEP: 
https://www.python.org/dev/peps/pep-0453/#implementation-strategy

The default presence of `setuptools` and `wheel` in `venv` created virtual 
environments at all is also an implementation detail that's likely to change 
once PEP 517 is implemented (again, as per the original design in PEP 453).

Emily: for 3.5+, venv is recommended just because it avoids the extra 3rd party 
dependency, and integrates slightly more nicely with the interpreter internals. 
As soon as you need greater assurances of cross-version consistency though, 
virtualenv is the better choice. (Donald Stufft actually has a virtualenv 
refactoring pending that makes it just a thin shell around venv on 3.5+, but 
pursuing that's on hold for now due to other priorities)

For the auto-upgrade aspect, environment managers like `pew` offer more 
flexibility to automate that kind of thing in a way that's consistent across 
Python versions. It *might* make sense to offer a `--upgrade-pip` flag for 
`venv` creation, but it's not entirely clear how much benefit that really 
offers over just running `python -m pip install --upgrade pip` inside the venv 
yourself. That one's probably worth filing as a separate RFE though, as it's 
probably the single most common post-creation script for new virtual 
environments.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30628] why venv install old pip?

[Curtis Doty]

Curtis Doty added the comment:

diff --git a/Lib/venv/__init__.py b/Lib/venv/__init__.py
index 716129d139..757e8de4e7 100644
--- a/Lib/venv/__init__.py
+++ b/Lib/venv/__init__.py
@@ -243,6 +243,9 @@ class EnvBuilder:
 cmd = [context.env_exe, '-Im', 'ensurepip', '--upgrade',
 '--default-pip']
 subprocess.check_output(cmd, stderr=subprocess.STDOUT)
+cmd= [context.env_exe, '-Im', 'pip', 'install', '--upgrade',
+'pip', 'setuptools']
+subprocess.check_output(cmd, stderr=subprocess.STDOUT)
 
 def setup_scripts(self, context):
 """

Crude, but effective. Food for thought?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30628] why venv install old pip?

[Emily Morehouse]

Emily Morehouse added the comment:

A quick note on how to include system packages in the "How Do I..?" section 
could be helpful, though does not necessarily solve the current issue. I would 
be hesitant to instruct people to use the system-site-packages flag simply to 
have an updated version of setuptools, even with a strong word of caution. From 
my experience, most people generally don't have a clean pip environment outside 
of a virtual environment for various reasons and this could cause more 
confusion and undesired outcomes by encouraging use use of the 
system-site-packages flag.

As a side note, I do feel that the existing documentation for the 
system-site-packages and without-pip flags is adequate as-is (that was one of 
the first things I tried when debugging this issue).

Nick, could you answer a couple of questions for me? 
1) I've read that (for Python 3.5 or greater) that venv is recommended over 
virtualenv. Why? (Not challenging anything, just trying to understand from the 
perspective of someone who has used virtualenv for years and never looked back).
2) Could there be an additional argument for venv that triggers an upgrade to 
setuptools on virtual environment creation? Alternatively, I've looked into 
extending venv.EnvBuilder. We could add an example of extending this to 
bootstrap environment building to include a setuptools upgrade on creation for 
those who deem this important.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30628] why venv install old pip?

[Nick Coghlan]

Nick Coghlan added the comment:

Emily's answer sort of covered the "It's not as simple as it might seem" 
aspect, whereby the bit that's genuinely tricky is to provide access to the 
system pip(/setuptools/wheel) without providing accessing to the system site 
packages in general.

If the latter is actually considered desirable, then the most suitable venv 
configuration is likely to be:

python3 -m venv --system-site-packages --without-pip

as that will give you a fresh venv to install your own additions into, but use 
the system pip package by default, rather than automatically bootstrapping a 
venv specific instance of it.

We don't currently discuss that kind of hybrid configuration in the docs, so it 
may be appropriate to reopen this as a docs issue, and add a new "How do I use 
system level Python packages from a virtual environment?" question to 
https://docs.python.org/3/installing/#how-do-i

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30628] why venv install old pip?

[R. David Murray]

R. David Murray added the comment:

OK, so the key here is venv's dependency on a pip wheel.  That makes sense, but 
is certainly counterintuitive.  One would naively think that having a copy of 
PIP installed would make it easy to link that into the venv, but obviously it 
is a lot more complex than that, given pip's dependencies.  Thanks for the 
explanation.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30628] why venv install old pip?

[Nick Coghlan]

Nick Coghlan added the comment:

The upstream supported install path for bootstrapping the venv is from a wheel 
file. The CPython source tree contains the necessary wheels, and, by default, 
upgrading the system pip doesn't touch them.

The only distro I can speak authoritatively for when it comes to doing things 
differently is Fedora, and what we do is:

- add a circular dependency between the system Python & the system pip
- add a circular dependency between the system Python & Fedora's "rewheel" 
utility
- patch CPython to remove the bundled wheel files
- patch ensurepip to "rewheel" the system pip and install that into freshly 
bootstrapped virtual environments

The end result works, but it really isn't pretty from a distro maintenance 
perspective, so not every distro is going to want to do it (Fedora has the 
luxury of Red Hat's Python maintenance team also maintaining the Fedora system 
Python).

It also isn't currently practical for upstream to make it work that way by 
default due to:

* the implied external dependency on the system pip (which not every distro is 
going to have)
* the need to recreate a wheel from an already installed package (which isn't 
officially supported by distutils-sig as a valid thing to do)
* the fact some distro's unbundle pip's vendored dependencies, so rewheeling 
just pip isn't necessarily sufficient (it is on Fedora, but that's only because 
we leave the vendored dependencies in place and trust pip to issue new updates 
promptly when necessary)
* the fact that not every distro lays down the distutils-metadata needed to get 
tools like rewheel to work (Debian's equivalent, dirtbike, uses deb metadata 
instead)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30628] why venv install old pip?

[R. David Murray]

R. David Murray added the comment:

It's not obvious from your discussion, Nick, *why* venv won't use an upgraded 
system pip if it has been manually upgraded.  There's no need for internet 
access in that case (which is the argument for using the bundled pip when 
running ensurepip for the first time).  For that matter, there's no obvious 
relationship between venv and ensurepip, other than ensurepip making sure there 
is at least one version of pip for venv to use.

So I agree with the OP that this behavior is quite surprising.  As well as 
annoying.  Perhaps there is a good reason for it that I'm not seeing?

--
nosy: +r.david.murray

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30628] why venv install old pip?

[Nick Coghlan]

Nick Coghlan added the comment:

Indeed, Emily is correct: this is expected behaviour for system Python versions 
that don't patch their ensurepip modules, as the default pip (et al) used in 
virtual environments created with the venv module generally only gets upgraded 
when upgrading to a new maintenance release of CPython.

Some redistributors do patch the ensurepip module to seed virtual environments 
based on the installed system level packages, so if your redistributor is one 
that already does so (e.g. Fedora), then you may have a downstream bug to file, 
and if they don't, then you may have a downstream feature request to ask them 
to switch to that model rather than using the files bundled by upstream.

In the meantime, the two main approaches folks adopt to make sure that they 
always have the latest versions of these components installed by default are:

- using the latest version of the third party `virtualenv` module for venv 
management (as that gets updated independently of the CPython maintenance 
release cycle)
- running "python -m pip install --upgrade pip setuptools wheel" immediately 
after creating each virtual environment

--
resolution:  -> not a bug
stage:  -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30628] why venv install old pip?

[Emily Morehouse]

Emily Morehouse added the comment:

You're certainly right. venv uses ensurepip to install pip when creating a 
virtual environment and does not access the internet/upgrade any packages. 
ensurepip was specifically designed to use the bundled version of setuptools. 
The bundled version of setuptools was up to date as of the final Python 3.6.0 
beta, though it seems like it could be due for an upgrade.

For now, you can either run `pip install --upgrade setuptools` after activating 
your environment (recommended) or give your environment access to your local 
site packages that has the upgraded setuptools using `python3 -m venv foo 
--system-site-packages` (not recommended, as this will include ALL global 
packages you have installed). Alternatively, virtualenv 
(https://virtualenv.pypa.io/en/stable/) maintains an updated version of 
setuptools more frequently, though it must be installed as an additional 
package.

It should also be noted that wheel is not installed by venv and must also be 
installed after the environment creation if needed.

--
nosy: +ncoghlan

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30628] why venv install old pip?

[Emily Morehouse]

Changes by Emily Morehouse :


--
nosy: +emilyemorehouse

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30628] why venv install old pip?

[Curtis Doty]

Curtis Doty added the comment:

$ pip3.6 list
pip (9.0.1)
setuptools (32.2.0)
wheel (0.29.0)

$ pip3.6 install --upgrade pip setuptools wheel
$ pip3.6 list
pip (9.0.1)
setuptools (36.0.1)
wheel (0.29.0)

$ python3 -m venv foo
$ source foo/bin/activate
(foo) $ pip list
pip (9.0.1)
setuptools (28.8.0)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30628] why venv install old pip?

[Curtis Doty]

New submission from Curtis Doty:

I've already updated system setuptools 36.0.1 with `pip3 install --upgrade pip 
setuptools wheel` but when I create a new enviro with `python3 -m venv foo` it 
will have an old/outdated setuptools 28.8.0 from embedded ensurepip.

Why? I think it should install the already-updated one!

--
components: Library (Lib)
messages: 295678
nosy: GreenKey
priority: normal
severity: normal
status: open
title: why venv install old pip?
type: behavior
versions: Python 3.6

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com