On 26. 05. 22 17:13, Owen Taylor wrote:
Thanks for the detailed response!
To start with your final questions:
> - Why is Python itself not installed in the /app prefix?
> - Where and how is pip install invoked?
The way that Flatpaks work is that there are *two* filesystem images, one
mounted on /usr and one mounted on /app. The image mounted on /usr is called
the "runtime" and contains libraries and other files shared between different
applications. In Fedora, this is made out of standard distribution packages -
glibc, libjpeg, and so forth. It also contains Python 3 and some modules.
The filesystem image at /app contains packages specific to a specific
application - the application itself and libraries. In Fedora, we create the
/app filesystem by rebuilding packages inside a module that is configured to
have a special RPM macros package installed in the buildroot -
https://src.fedoraproject.org/rpms/flatpak-rpm-macros/tree/f36
<https://src.fedoraproject.org/rpms/flatpak-rpm-macros/tree/f36> (installed at
/etc/rpm/macros.flatpak so that it has higher priority than the macros from
/usr/lib/rpm)
(The advantage of this split over the docker layer system is that applications
don't need to be rebuilt every time the base layer is updated.)
macros.flatpak redefines %_prefix to /app, and then does various more and less
hacky things to make that actual work within the Fedora RPM ecosystem - in
particular for Python, it overrides
%python3_sitelib and %python3_sitearch. The distutils.cfg I mentioned is
installed from the same package. This combination worked pretty well for F35
for Python.
pip install is being invoked is by %pyproject_install or %py_install_wheel out
of an RPM build - the particular breakage that triggered this investigation was
the Python bindings for zxing-cpp , but I think it would apply to pretty much
anything that has moved to the %pyproject macros.
In that case I suppose we could change those macros to pass --prefix %{_prefix}
explicitly. [I thought this before reading the rest of your emial, keeping it
here.]
To read what changed in Fedora 36, I suggest reading the release notes:
https://docs.fedoraproject.org/en-US/fedora/latest/release-notes/developers/Development_Python/#sect-install_scheme
<https://docs.fedoraproject.org/en-US/fedora/latest/release-notes/developers/Development_Python/#sect-install_scheme>
And this email thread:
https://lists.fedoraproject.org/archives/list/python-devel@lists.fedoraproject.org/thread/AAGUFQZ4RZDU7KUN4HA43KQJCMSFR3GW/
<https://lists.fedoraproject.org/archives/list/python-devel@lists.fedoraproject.org/thread/AAGUFQZ4RZDU7KUN4HA43KQJCMSFR3GW/>
When we did this, we have dropped sysconfig._PIP_USE_SYSCONFIG in Fedora
36+.
The change in our patching also caused a naughty UX-nightmare:
https://bugzilla.redhat.com/show_bug.cgi?id=2026979
<https://bugzilla.redhat.com/show_bug.cgi?id=2026979>
However, when $RPM_BUILD_ROOT is set, this should not matter. As long as you
call `pip install` from the specfile (how *exactly* are you using pip
install?)
you should be not impacted (much). I am just mentioning this for
completeness,
as I am unaware of how you use pip exactly.
The fact that sysconfig behaves differently when $RPM_BUILD_ROOT is set wasn't
something I was expecting, and explains quite a bit :-)
I have dedicated my time in the upcoming months to try to rework this
entirely
for Fedora 37+, as it seems I rather screwed this up in the effort to try to
make it better. The new approach will however also patch sysconfig and not
distutils. Using distutils for anything has no future.
(My new idea is about distro-customizable default sysconfig installation
prefix
value, and I am happy to brag about it more -- we could possibly use it as a
future-proof solution for /app as well, depending on what you want to
achieve.)
Having some standard way to configure a system by environment variable or drop
in file would definitely be nice ... and sounds like something we could leverage.
Awesome.
Though I'm also wondering now if we could just change %py_install variants and
%pyproject variants to have --prefix %{_prefix} on the install command line -
that should have no effect for standard builds and make Flatpak builds work?
Now I see you are in agreement with me previous thought. That should indeed
work.
As a hotfix, you could probably set sysconfig._PIP_USE_SYSCONFIG back to
False.
RPM-packaged pip in Fedora 36 is 21.3 so we would need to backport my change
that reads this value to make it work (I have no problem doing that).
Those are the ideas that come to my mind about how to set that attribute:
- build your own forked Python package that sets this
- build your own forked pip package that defaults to this
We definitely have the *ability* to use a stream branch for python3 or
python-pip, and build it in the flatpak-runtime package. But I'd really prefer
to avoid that, since that creates a fork we need to maintain every time that
these packages are updated in Fedora 36, and there's a possibility of missing
security updates.
Makes sense. I kinda suspected that, that's why I proposed the next option.
- set it via a Python module that is imported from
/usr/lib/python3.10/site-packages/_pip_use_sysconfig.pth
However, forcing pip to use distutils is the opposite of future-proof.
When looking for a permanent solution, you need to avoid distutils entirely.
A pip with a backported patch (or a rebase) + this approach sounds most
maintainable for Fedora 36. But if we think that the --prefix %{_prefix}
approach actually works, then perhaps we can either:
A) Just update python-rpm-macros and pygobject-rpm-macros in F36
B) Build a stream-branch (or even the f37 branch) in the flatpak-runtime
module for F36 and reconverge for F37 - I'm less worried about carrying a
stream branch for macros packages, since they are unlikely to have security fixes.
What do you think?
Definitively A. There is no reason to fork the macros, the change should be
perfectly backwards compatible.
The lines are:
https://src.fedoraproject.org/rpms/python-rpm-macros/blob/f36/f/macros.python#_61
https://src.fedoraproject.org/rpms/python-rpm-macros/blob/f36/f/macros.python3#_59
https://src.fedoraproject.org/rpms/pyproject-rpm-macros/blob/rawhide/f/macros.pyproject#_57
--
Miro Hrončok
--
Phone: +420777974800
IRC: mhroncok
_______________________________________________
python-devel mailing list -- python-devel@lists.fedoraproject.org
To unsubscribe send an email to python-devel-le...@lists.fedoraproject.org
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/python-devel@lists.fedoraproject.org
Do not reply to spam on the list, report it:
https://pagure.io/fedora-infrastructure
