Action required
===============
I would suggest to raise the severity of this bug.
Reason
======
This is a major blocker for using Debian as an "integrated Python
distribution"[1]
since you cannot create a venv for (probably) ANY of the older Python releases.
Creating older envs is an essential task for today's developers when
testing & evaluating code.
The issue arises in 2 setups:
- during the transition periods between Python-releases,
where both new and old packages exist in the same apt-repositories
(as currently in unstable(sid), or
- (i guess) when installing older pythons from "deadsnakes" repo.
The message received is the typical debian-workaround message included
in `libpython3.*-stdlib` packages,
which unfortunately masquerades the real cause (will file a separate
bug for this).
For instance in "sid":
$ python3.8 -m venv .venv
The virtual environment was not created successfully because
ensurepip is not
available. On Debian/Ubuntu systems, you need to install the python3-venv
package using the following command.
apt-get install python3-venv
You may need to use sudo with that command. After installing the
python3-venv
package, recreate your virtual environment.
Failing command: ['/home/user/.venv/bin/python3.8', '-Im',
'ensurepip', '--upgrade', '--default-pip']
It's only when you try the failing command that you get the root cause:
$ /home/user/.venv/bin/python3.8 -Im ensurepip --upgrade --default-pip
Traceback (most recent call last):
File "<string>", line 6, in <module>
File "/usr/lib/python3.8/runpy.py", line 207, in run_module
return _run_module_code(code, init_globals, run_name, mod_spec)
File "/usr/lib/python3.8/runpy.py", line 97, in _run_module_code
_run_code(code, mod_globals, init_globals,
File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/tmp/tmpybc1t180/pip-20.1.1-py2.py3-none-any.whl/pip/__main__.py",
line 23, in <module>
File "<frozen zipimport>", line 259, in load_module
File
"/tmp/tmpybc1t180/pip-20.1.1-py2.py3-none-any.whl/pip/_internal/cli/main.py",
line 10, in <module>
File "<frozen zipimport>", line 259, in load_module
File
"/tmp/tmpybc1t180/pip-20.1.1-py2.py3-none-any.whl/pip/_internal/cli/autocompletion.py",
line 9, in <module>
File "<frozen zipimport>", line 259, in load_module
File
"/tmp/tmpybc1t180/pip-20.1.1-py2.py3-none-any.whl/pip/_internal/cli/main_parser.py",
line 7, in <module>
File "<frozen zipimport>", line 259, in load_module
File
"/tmp/tmpybc1t180/pip-20.1.1-py2.py3-none-any.whl/pip/_internal/cli/cmdoptions.py",
line 19, in <module>
ModuleNotFoundError: No module named 'distutils.util'
Traceback (most recent call last):
File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/usr/lib/python3.8/ensurepip/__main__.py", line 5, in <module>
sys.exit(ensurepip._main())
File "/usr/lib/python3.8/ensurepip/__init__.py", line 265, in _main
return _bootstrap(
File "/usr/lib/python3.8/ensurepip/__init__.py", line 183, in _bootstrap
return _run_pip(args + _PROJECTS, additional_paths)
File "/usr/lib/python3.8/ensurepip/__init__.py", line 61, in _run_pip
return subprocess.run([sys.executable, "-c", code],
check=True).returncode
File "/usr/lib/python3.8/when trying the", in run
raise the real cause CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command
'['/home/user/.venv/bin/python3.8', '-c', '\nimport runpy\nimport
sys\nsys.path =
[\'/tmp/tmpybc1t180/setuptools-44.0.0-py2.py3-none-any.whl\',
\'/tmp/tmpybc1t180/pip-20.1.1-py2.py3-none-any.whl\',
\'/tmp/tmpybc1t180/pkg_resources-0.0.0-py2.py3-none-any.whl\'] +
sys.path\nsys.argv[1:] = [\'install\', \'--no-cache-dir\',
\'--no-index\', \'--find-links\', \'/tmp/tmpybc1t180\', \'--upgrade\',
\'setuptools\', \'pip\', \'pkg_resources\']\nrunpy.run_module("pip",
run_name="__main__", alter_sys=True)\n']' returned non-zero exit
status 1.
The reason for the `ModuleNotFoundError` in both cases is because,
for example in "sid", /usr/lib/python3.8/distutils/ contains just
dummy package files
from `libpython3.8-stdlib`
and `python3-distutils` installs the real package only in
/usr/lib/python3.9/distutils/,
as Shai Berger explained above.
Workaround
==========
There is a SO article[2] that suggests to copy all files from:
/usr/lib/python3.9/distutils --> /usr/lib/python3.8/distutils
I would suggest that a simple symlink would suffice:
sudo rm -r /usr/lib/python3.8/distutils/
sudo ln -s /usr/lib/python3.{9,8}/distutils
BUT(!) this trick which used to work for python3.7-->3.8 transition,
STOP WORKING for python3.8-->3.9 transition (e.g. "sid").
If you rerun the failing command, above, after the symlink has been placed,
you get:
$ /home/user/.venv/bin/python3.8 -Im ensurepip --upgrade --default-pip
Looking in links: /tmp/tmpsawo7ko_
Processing /tmp/tmpsawo7ko_/setuptools-44.0.0-py2.py3-none-any.whl
Processing /tmp/tmpsawo7ko_/pip-20.1.1-py2.py3-none-any.whl
Processing /tmp/tmpsawo7ko_/pkg_resources-0.0.0-py2.py3-none-any.whl
Installing collected packages: setuptools, pip, pkg-resources
ERROR: Exception:
Traceback (most recent call last):
File
"/tmp/tmpsawo7ko_/pip-20.1.1-py2.py3-none-any.whl/pip/_internal/cli/base_command.py",
line 188, in _main
status = self.run(options, args)
File
"/tmp/tmpsawo7ko_/pip-20.1.1-py2.py3-none-any.whl/pip/_internal/cli/req_command.py",
line 185, in wrapper
return func(self, options, args)
File
"/tmp/tmpsawo7ko_/pip-20.1.1-py2.py3-none-any.whl/pip/_internal/commands/install.py",
line 424, in run
installed = install_given_reqs(
File
"/tmp/tmpsawo7ko_/pip-20.1.1-py2.py3-none-any.whl/pip/_internal/req/__init__.py",
line 67, in install_given_reqs
requirement.install(
File
"/tmp/tmpsawo7ko_/pip-20.1.1-py2.py3-none-any.whl/pip/_internal/req/req_install.py",
line 769, in install
scheme = get_scheme(
File
"/tmp/tmpsawo7ko_/pip-20.1.1-py2.py3-none-any.whl/pip/_internal/locations.py",
line 191, in get_scheme
scheme = distutils_scheme(
File
"/tmp/tmpsawo7ko_/pip-20.1.1-py2.py3-none-any.whl/pip/_internal/locations.py",
line 131, in distutils_scheme
i.finalize_options()
File "/usr/lib/python3.8/distutils/command/install.py", line
306, in finalize_options
(prefix, exec_prefix) = get_config_vars('prefix', 'exec_prefix')
File "/usr/lib/python3.8/distutils/sysconfig.py", line 541, in
get_config_vars
srcdir = os.path.dirname(get_makefile_filename())
File "/usr/lib/python3.8/distutils/sysconfig.py", line 289, in
get_makefile_filename
lib_dir = get_python_lib(plat_specific=0, standard_lib=1)
File "/usr/lib/python3.8/distutils/sysconfig.py", line 156, in
get_python_lib
libdir = sys.platlibdir
AttributeError: module 'sys' has no attribute 'platlibdir'
Traceback (most recent call last):
File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/usr/lib/python3.8/ensurepip/__main__.py", line 5, in <module>
sys.exit(ensurepip._main())
File "/usr/lib/python3.8/ensurepip/__init__.py", line 265, in _main
return _bootstrap(
File "/usr/lib/python3.8/ensurepip/__init__.py", line 183, in _bootstrap
return _run_pip(args + _PROJECTS, additional_paths)
File "/usr/lib/python3.8/ensurepip/__init__.py", line 61, in _run_pip
return subprocess.run([sys.executable, "-c", code],
check=True).returncode
File "/usr/lib/python3.8/subprocess.py", line 512, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command
'['/home/user/.venv/bin/python3.8', '-c', '\nimport runpy\nimport
sys\nsys.path =
[\'/tmp/tmpsawo7ko_/setuptools-44.0.0-py2.py3-none-any.whl\',
\'/tmp/tmpsawo7ko_/pip-20.1.1-py2.py3-none-any.whl\',
\'/tmp/tmpsawo7ko_/pkg_resources-0.0.0-py2.py3-none-any.whl\'] +
sys.path\nsys.argv[1:] = [\'install\', \'--no-cache-dir\',
\'--no-index\', \'--find-links\', \'/tmp/tmpsawo7ko_\', \'--upgrade\',
\'setuptools\', \'pip\', \'pkg_resources\']\nrunpy.run_module("pip",
run_name="__main__", alter_sys=True)\n']' returned non-zero exit
status 2.
The workaround fails for python3.8 because the `distutils` in python3.9 is using
the new `sys.platlibdir` attribute[3],
so somehow we have to place there the original files for distutils-3.8.
Any ideas?
References:
===========
[1]: https://wiki.debian.org/Python#Python_in_Debian
[2]: https://askubuntu.com/a/1260519/251379
[3]: https://docs.python.org/3/library/sys.html#sys.platlibdir
