Thanks for the detailed response! To start with your final questions:
> - Why is Python itself not installed in the /app prefix? > - Where and how is pip install invoked? The way that Flatpaks work is that there are *two* filesystem images, one mounted on /usr and one mounted on /app. The image mounted on /usr is called the "runtime" and contains libraries and other files shared between different applications. In Fedora, this is made out of standard distribution packages - glibc, libjpeg, and so forth. It also contains Python 3 and some modules. The filesystem image at /app contains packages specific to a specific application - the application itself and libraries. In Fedora, we create the /app filesystem by rebuilding packages inside a module that is configured to have a special RPM macros package installed in the buildroot - https://src.fedoraproject.org/rpms/flatpak-rpm-macros/tree/f36 (installed at /etc/rpm/macros.flatpak so that it has higher priority than the macros from /usr/lib/rpm) (The advantage of this split over the docker layer system is that applications don't need to be rebuilt every time the base layer is updated.) macros.flatpak redefines %_prefix to /app, and then does various more and less hacky things to make that actual work within the Fedora RPM ecosystem - in particular for Python, it overrides %python3_sitelib and %python3_sitearch. The distutils.cfg I mentioned is installed from the same package. This combination worked pretty well for F35 for Python. pip install is being invoked is by %pyproject_install or %py_install_wheel out of an RPM build - the particular breakage that triggered this investigation was the Python bindings for zxing-cpp , but I think it would apply to pretty much anything that has moved to the %pyproject macros. On Thu, May 26, 2022 at 4:04 AM Miro Hrončok <mhron...@redhat.com> wrote: [...] > To read what changed in Fedora 36, I suggest reading the release notes: > > > https://docs.fedoraproject.org/en-US/fedora/latest/release-notes/developers/Development_Python/#sect-install_scheme > > And this email thread: > > > https://lists.fedoraproject.org/archives/list/python-devel@lists.fedoraproject.org/thread/AAGUFQZ4RZDU7KUN4HA43KQJCMSFR3GW/ > > When we did this, we have dropped sysconfig._PIP_USE_SYSCONFIG in Fedora > 36+. > > The change in our patching also caused a naughty UX-nightmare: > https://bugzilla.redhat.com/show_bug.cgi?id=2026979 > > However, when $RPM_BUILD_ROOT is set, this should not matter. As long as > you > call `pip install` from the specfile (how *exactly* are you using pip > install?) > you should be not impacted (much). I am just mentioning this for > completeness, > as I am unaware of how you use pip exactly. > The fact that sysconfig behaves differently when $RPM_BUILD_ROOT is set wasn't something I was expecting, and explains quite a bit :-) > I have dedicated my time in the upcoming months to try to rework this > entirely > for Fedora 37+, as it seems I rather screwed this up in the effort to try > to > make it better. The new approach will however also patch sysconfig and not > distutils. Using distutils for anything has no future. > > (My new idea is about distro-customizable default sysconfig installation > prefix > value, and I am happy to brag about it more -- we could possibly use it as > a > future-proof solution for /app as well, depending on what you want to > achieve.) > Having some standard way to configure a system by environment variable or drop in file would definitely be nice ... and sounds like something we could leverage. Though I'm also wondering now if we could just change %py_install variants and %pyproject variants to have --prefix %{_prefix} on the install command line - that should have no effect for standard builds and make Flatpak builds work? As a hotfix, you could probably set sysconfig._PIP_USE_SYSCONFIG back to > False. > RPM-packaged pip in Fedora 36 is 21.3 so we would need to backport my > change > that reads this value to make it work (I have no problem doing that). > > Those are the ideas that come to my mind about how to set that attribute: > > - build your own forked Python package that sets this > - build your own forked pip package that defaults to this > We definitely have the *ability* to use a stream branch for python3 or python-pip, and build it in the flatpak-runtime package. But I'd really prefer to avoid that, since that creates a fork we need to maintain every time that these packages are updated in Fedora 36, and there's a possibility of missing security updates. > - set it via a Python module that is imported from > /usr/lib/python3.10/site-packages/_pip_use_sysconfig.pth > > However, forcing pip to use distutils is the opposite of future-proof. > When looking for a permanent solution, you need to avoid distutils > entirely. > A pip with a backported patch (or a rebase) + this approach sounds most maintainable for Fedora 36. But if we think that the --prefix %{_prefix} approach actually works, then perhaps we can either: A) Just update python-rpm-macros and pygobject-rpm-macros in F36 B) Build a stream-branch (or even the f37 branch) in the flatpak-runtime module for F36 and reconverge for F37 - I'm less worried about carrying a stream branch for macros packages, since they are unlikely to have security fixes. What do you think? - Owen
_______________________________________________ python-devel mailing list -- python-devel@lists.fedoraproject.org To unsubscribe send an email to python-devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/python-devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
