Hello Pythonistas.
tl;dr I wonder if we should get rid of the last downstream-only patch in
Fedora's Python,
the one responsible for /usr/local/lib(64)/python... installation location.
The removal would bring us closer to upstream, but perhaps cause a regression
for our users.
I propose to adapt PEP 668 (Marking Python base environments as “externally
managed”) instead.
Details:
On Python 3.13+ we only ship one downstream-only patch in Python.
That is a patch that is not yet upstreamable, isn't a backport etc.
The patch is this:
https://src.fedoraproject.org/rpms/python3.13/blob/f39/f/00251-change-user-install-location.patch
(Deliberately linking f39 to make the link stable. Rawhide is currently the
same.)
Or as a commit:
https://github.com/fedora-python/cpython/commit/ac3a9df5f8
We carry this patch in various forms for 7+ years.
It originated with https://fedoraproject.org/wiki/Changes/Making_sudo_pip_safe
The idea is this:
- sudo pip install etc. installs to /usr/local/lib(64)/python3.X/site-packages
- RPM packages are installed in /usr/lib(64)/python3.X/site-packages
- RPM installed software ignores the packages in /usr/local (unless they opt
in not to)
- other software uses packages from both, giving preference to /usr/local
There are several challenges to this approach:
One of them is upgrading. Ideally if pyfoo 1 is installed via RPM,
sudo pip install --upgrade 'pyfoo>=2' should install pyfoo to /usr/local while
leaving
the RPM-installed pyfoo 1 be.
In reality, pip is not ready for this and needs a patch as well:
https://src.fedoraproject.org/rpms/python-pip/blob/f39/f/remove-existing-dist-only-if-path-conflicts.patch
This means that users who do sudo pip install --upgrade pip lose this patch.
This also means that the fact that this appears to work in other Python package
managers (such as uv) is probably purely accidental.
This behavior also tends to differ for RPM-packaged Python packages with
.dist-info and .egg-info metadata.
Another challenge is the mechanism we decided to use for RPM packages to ignore
/usr/local.
We use the -s flag in shebangs for this. That leads to problems like this:
"global pip installed modules not visible if user site dir is disabled"
https://bugzilla.redhat.com/2256190
We could probably solve this by introducing a custom flag instead of -s,
but that would make the patch even worse -- our RPM-packaged software would not
even execute
without our Python with our patch. I don't want to do that.
A result of this problem (which is shared with Debian-based distros) was
PEP 668 – Marking Python base environments as “externally managed”
https://peps.python.org/pep-0668/
Except we cannot use it as is, because our install locations share a common
stdlib location.
I described this at
https://discuss.python.org/t/pep-668-marking-python-base-environments-as-externally-managed/10302/46
and further.
At this point, after 2 years of failure, I really don't know how to get
something from this that's upstreamable.
I spoke about this at PyCon PL '22 as well:
https://www.youtube.com/watch?v=7SBH4PkbMhw
---
Recently it occurred to me that perhaps we could:
1. drop the patches from Python and pip
2. mark /usr/lib(64)/python3.X as externally managed (PEP 668)
And declare sudo-pip-installing unsupported. (That means, it only works with a
custom pip flag and if users nuke their systems by using that flag, they shot
themelves and we cannot help them.)
Personally, I think this would be the best solution:
sudo-pip-installing is bad and should never be done (no, not even in
containers).
However, I realize that others might disagree. For them, I can offer pip
--break-system-packages.
If we want to do this, I think we need to do this on Python upgrade boundary.
We don't want users sudo pip-installing packages with Python 3.13 on Fedora 41
only to find them
ignored on Fedora 42.
However, when we upgrade Python to 3.14 in Fedora 43, their packages would
still be broken anyway.
--
Miro Hrončok
--
Phone: +420777974800
Fedora Matrix: mhroncok
--
_______________________________________________
python-devel mailing list -- python-devel@lists.fedoraproject.org
To unsubscribe send an email to python-devel-le...@lists.fedoraproject.org
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/python-devel@lists.fedoraproject.org
Do not reply to spam, report it:
https://pagure.io/fedora-infrastructure/new_issue
