On 12/4/24 23:28, Charalampos Stratakis via python-devel wrote:
On Tue, Dec 3, 2024 at 7:02 PM Miro Hrončok via python-devel
<python-devel@lists.fedoraproject.org> wrote:
Hello Pythonistas.
tl;dr I wonder if we should get rid of the last downstream-only
patch in
Fedora's Python,
the one responsible for /usr/local/lib(64)/python... installation
location.
The removal would bring us closer to upstream, but perhaps cause a
regression
for our users.
I propose to adapt PEP 668 (Marking Python base environments as
“externally
managed”) instead.
Details:
On Python 3.13+ we only ship one downstream-only patch in Python.
That is a patch that is not yet upstreamable, isn't a backport etc.
The patch is this:
https://src.fedoraproject.org/rpms/python3.13/blob/f39/f/00251-change-user-install-location.patch
(Deliberately linking f39 to make the link stable. Rawhide is
currently the same.)
Or as a commit:
https://github.com/fedora-python/cpython/commit/ac3a9df5f8
We carry this patch in various forms for 7+ years.
It originated with
https://fedoraproject.org/wiki/Changes/Making_sudo_pip_safe
The idea is this:
- sudo pip install etc. installs to
/usr/local/lib(64)/python3.X/site-packages
- RPM packages are installed in /usr/lib(64)/python3.X/site-packages
- RPM installed software ignores the packages in /usr/local
(unless they opt
in not to)
- other software uses packages from both, giving preference to
/usr/local
There are several challenges to this approach:
One of them is upgrading. Ideally if pyfoo 1 is installed via RPM,
sudo pip install --upgrade 'pyfoo>=2' should install pyfoo to
/usr/local while
leaving
the RPM-installed pyfoo 1 be.
In reality, pip is not ready for this and needs a patch as well:
https://src.fedoraproject.org/rpms/python-pip/blob/f39/f/remove-existing-dist-only-if-path-conflicts.patch
This means that users who do sudo pip install --upgrade pip lose
this patch.
This also means that the fact that this appears to work in other
Python package
managers (such as uv) is probably purely accidental.
This behavior also tends to differ for RPM-packaged Python
packages with
.dist-info and .egg-info metadata.
Another challenge is the mechanism we decided to use for RPM
packages to ignore
/usr/local.
We use the -s flag in shebangs for this. That leads to problems
like this:
"global pip installed modules not visible if user site dir is
disabled"
https://bugzilla.redhat.com/2256190
We could probably solve this by introducing a custom flag instead
of -s,
but that would make the patch even worse -- our RPM-packaged
software would not
even execute
without our Python with our patch. I don't want to do that.
A result of this problem (which is shared with Debian-based
distros) was
PEP 668 – Marking Python base environments as “externally managed”
https://peps.python.org/pep-0668/
Except we cannot use it as is, because our install locations share
a common
stdlib location.
I described this at
https://discuss.python.org/t/pep-668-marking-python-base-environments-as-externally-managed/10302/46
and further.
At this point, after 2 years of failure, I really don't know how
to get
something from this that's upstreamable.
I spoke about this at PyCon PL '22 as well:
https://www.youtube.com/watch?v=7SBH4PkbMhw
---
Recently it occurred to me that perhaps we could:
1. drop the patches from Python and pip
2. mark /usr/lib(64)/python3.X as externally managed (PEP 668)
So the implication here is that pip will not be allowed to install a
package in the system directories that RPMs are installed in?
And declare sudo-pip-installing unsupported. (That means, it only
works with a
custom pip flag and if users nuke their systems by using that
flag, they shot
themelves and we cannot help them.)
Honestly I don't see this working and I believe that ship sailed a
long time ago unfortunately. Way too many tutorials, books and even
workshops recommend "sudo pip install" and before our patch we always
had a constant stream of bugs filled about users nuking their systems
that way.
Hi all,
I agree with that. This is a righteous fight, but one we're going to
lose. If we "break" `sudo pip install`, we're going to break a lot of
people's (maybe wrong) use cases, gonna get a lot of angry and/or
confused users, and Fedora might not be seen as loving Python so much
anymore.
On top of that, I really worry that if our containers don't support sudo
pip, we're going to lose users, because our competitors do support this
common use case. I also talked to the PM for Python in RHEL, and he
believes sudo pip must work in RHEL containers (that is of course not
binding for this Fedora discussion though).
I fully agree that the status quo is not great. But I'm not convinced
this change would make it better, especially not from the users'
perspective.
One thing that comes to my mind is that we don't really need any safe
guards in containers, because if you break a container with sudo pip,
you just restart it and fix your install script. So perhaps, we could
mark Python base environments as externally managed only in live
installs, and keep it free in containers, which would solve part of the
problem. On the other hand, it's not great to have a different container
environment, and could lead to unforeseen issues.
Tomas
Personally, I think this would be the best solution:
sudo-pip-installing is bad and should never be done (no, not even
in containers).
However, I realize that others might disagree. For them, I can
offer pip
--break-system-packages.
If we want to do this, I think we need to do this on Python
upgrade boundary.
We don't want users sudo pip-installing packages with Python 3.13
on Fedora 41
only to find them
ignored on Fedora 42.
However, when we upgrade Python to 3.14 in Fedora 43, their
packages would
still be broken anyway.
--
Miro Hrončok
--
Phone: +420777974800
Fedora Matrix: mhroncok
--
_______________________________________________
python-devel mailing list -- python-devel@lists.fedoraproject.org
To unsubscribe send an email to
python-devel-le...@lists.fedoraproject.org
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines:
https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/python-devel@lists.fedoraproject.org
Do not reply to spam, report it:
https://pagure.io/fedora-infrastructure/new_issue
--
Regards,
Charalampos Stratakis
Senior Software Engineer
Python Maintenance Team, Red Hat
--
_______________________________________________
python-devel mailing list -- python-devel@lists.fedoraproject.org
To unsubscribe send an email to python-devel-le...@lists.fedoraproject.org
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/python-devel@lists.fedoraproject.org
Do not reply to spam, report it:
https://pagure.io/fedora-infrastructure/new_issue
