On 26. 05. 22 6:10, Owen Taylor wrote:
[ At Miro's request, resending this to python-devel so the discussion can be
public ]
Hi Miro -
Hey Owen.
[the same here]
When rebuilding a package to include in a Flatpak, we want to install
*everything* under prefix=/app - that includes Python modules. (Paths will be
adjusted properly when running the Flatpak.)
For years, the way we did that is by installing a
/usr/lib64/python3.10/distutils/distutils.cfg:
===
[install]
prefix=/app
===
This is part of the flatpak-rpm-macros package that gets installed in the
buildroot for Flatpak rebuilds
In Fedora 36, this no longer works for 'pip install' - it seems that pip
unconditionally uses 'sysconfig' rather than 'distutils' to determine install
paths for Python >= 3.10.
I see that you filed https://github.com/pypa/pip/issues/10647
<https://github.com/pypa/pip/issues/10647> which landed in pip for pip-22, but
Fedora 36 / pip-21 this isn't present? And in any case, that seems like it's
something for 'sudo pip install' rather than package installation. I actually
don't really follow how the Python macros are getting things installed into
/usr/lib rather than /usr/local/lib.
Any advice about how we can make the installation into /app work? Hopefully in
a future-proof way...
Since pip 21.3, pip uses sysconfig rather than distutils (but only on Python
3.10+).
As we used to patch distutils in Fedora 35, we needed pip to use distutils
there. When pip made this change it was too late for us to change our patches
significantly to patch sysconfig instead. I've opened the issue you linked that
allowed us to *delay* this pip change in Fedora 35. And we did. See
https://src.fedoraproject.org/rpms/python3.10/blob/f35/f/00251-change-user-install-location.patch
- it has:
diff --git a/Lib/sysconfig.py b/Lib/sysconfig.py
+_PIP_USE_SYSCONFIG = False
Note that this was aimed purely for pip-installed or pip-upgraded pip (users
tend to do that). The RPM-installed pip in Fedora 35 was < 21.3 so it was not
affected at all and uses distutils.
As the Python ecosystem moves away from distutils we needed to stop patching it
and instead, reworked our patching significantly to patch syconfig instead.
This is needed because distutils is going to be removed and because pip won't
support our hack forever.
To read what changed in Fedora 36, I suggest reading the release notes:
https://docs.fedoraproject.org/en-US/fedora/latest/release-notes/developers/Development_Python/#sect-install_scheme
And this email thread:
https://lists.fedoraproject.org/archives/list/python-devel@lists.fedoraproject.org/thread/AAGUFQZ4RZDU7KUN4HA43KQJCMSFR3GW/
When we did this, we have dropped sysconfig._PIP_USE_SYSCONFIG in Fedora 36+.
The change in our patching also caused a naughty UX-nightmare:
https://bugzilla.redhat.com/show_bug.cgi?id=2026979
However, when $RPM_BUILD_ROOT is set, this should not matter. As long as you
call `pip install` from the specfile (how *exactly* are you using pip install?)
you should be not impacted (much). I am just mentioning this for completeness,
as I am unaware of how you use pip exactly.
I have dedicated my time in the upcoming months to try to rework this entirely
for Fedora 37+, as it seems I rather screwed this up in the effort to try to
make it better. The new approach will however also patch sysconfig and not
distutils. Using distutils for anything has no future.
(My new idea is about distro-customizable default sysconfig installation prefix
value, and I am happy to brag about it more -- we could possibly use it as a
future-proof solution for /app as well, depending on what you want to achieve.)
------
As a hotfix, you could probably set sysconfig._PIP_USE_SYSCONFIG back to False.
RPM-packaged pip in Fedora 36 is 21.3 so we would need to backport my change
that reads this value to make it work (I have no problem doing that).
Those are the ideas that come to my mind about how to set that attribute:
- build your own forked Python package that sets this
- build your own forked pip package that defaults to this
- set it via a Python module that is imported from
/usr/lib/python3.10/site-packages/_pip_use_sysconfig.pth
However, forcing pip to use distutils is the opposite of future-proof.
When looking for a permanent solution, you need to avoid distutils entirely.
To be able to suggest alternatives, I need to understand what it is that you
are doing.
- Why is Python itself not installed in the /app prefix?
- Where and how is pip install invoked?
--
Miro Hrončok
--
Phone: +420777974800
IRC: mhroncok
_______________________________________________
python-devel mailing list -- python-devel@lists.fedoraproject.org
To unsubscribe send an email to python-devel-le...@lists.fedoraproject.org
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/python-devel@lists.fedoraproject.org
Do not reply to spam on the list, report it:
https://pagure.io/fedora-infrastructure
