Bug#1060838: meson: should use the host gobject-introspection-1.0.pc for looking up g-ir-scanner
On Mon, 26 Feb 2024 at 17:58, Simon McVittie wrote: > I did consider that, but if I'm understanding what you propose > correctly, that would involve going through these steps for every > possible cross-tool: > > * inventing a new environment variable similar to CC and PKG_CONFIG, for > example perhaps G_IR_SCANNER in this case > * making debhelper (or dpkg) add it to the environment > * making `meson env2mfile` (or the older debcrossgen) read it from the > environment and add it to the generated cross-file That is not how it would work at all. Meson's env2mfile has a command line option that tells it to generate things for Debian package building. The idea is that then you can add any custom detection logic you could possibly want. In this way custom logic can be added in the main project in an actual programming language rather than trying to replicate the same via envvars and magic hacks.
Bug#1060838: meson: should use the host gobject-introspection-1.0.pc for looking up g-ir-scanner
On Sun, 25 Feb 2024 at 03:21:23 +0200, Jussi Pakkanen wrote: > Meson comes with a tool that converts native and cross environments > into cross files. It is run with `meson env2mfile`. It even has a mode > to generate Debian package building scripts. Updating that to handle > this case would probably be the smartest thing to do. I did consider that, but if I'm understanding what you propose correctly, that would involve going through these steps for every possible cross-tool: * inventing a new environment variable similar to CC and PKG_CONFIG, for example perhaps G_IR_SCANNER in this case * making debhelper (or dpkg) add it to the environment * making `meson env2mfile` (or the older debcrossgen) read it from the environment and add it to the generated cross-file and that seems like something that would scale poorly with the number of cross-tools that exist? We would have to coordinate changes between debhelper and meson every time a new cross-tool was identified. Or do you mean that `meson env2mfile` would have a long list of pkg-config queries that it would do using ${PKG_CONFIG} in order to populate the cross-file, like for example in this case, if gobject-introspection-1.0.pc was found, it would output the equivalent of the shell-like pseudocode "g-ir-scanner = '$(${PKG_CONFIG} --variable=g_ir_scanner gobject-introspection-1.0)'" and so on? That would avoid needing to invent new environment variables and modify debhelper, but it would still make updating `meson env2mfile` into a single point of failure for the ability to use any cross-tool. I see two possible routes that would scale better. One is what Helmut suggested: if running a cross-tool might make sense, ask the host pkg-config rather than the build pkg-config for the path to that tool, and rely on the packaging having been set up to make that work (as I already did for gobject-introspection). For the finite number of tools discovered internally by meson (like g-ir-scanner in the gnome module) this would still need to be a meson change, but for tools run by third-party packages as a custom_target or similar, it would be up to the third-party package to do this (although Meson could perhaps encourage it in documentation). The other is to do something with the common convention of an architecture prefix. In Autotools-world, there's a convention that if running a cross-tool might make sense, then the configure script looks for ${host_tuple}-${tool} before ${tool}, where ${tool} is something like pkg-config or g-ir-scanner, and ${host_tuple} is the host architecture given to the configure script (in Debian this is ${DEB_HOST_GNU_TYPE}). Similarly, when using plain Makefiles, it's relatively common to ask for ${CROSS_COMPILE}${tool}, where ${CROSS_COMPILE} is normally empty, but can be set to the host tuple followed by "-" when cross-compiling. Either of these will successfully find our /usr/bin/x86_64-linux-gnu-g-ir-scanner and so on. smcv
Bug#1060838: meson: should use the host gobject-introspection-1.0.pc for looking up g-ir-scanner
On Fri, 23 Feb 2024 at 14:27, Simon McVittie wrote: > On Wed, 21 Feb 2024 at 10:47:33 +0100, Johannes Schauer Marin Rodrigues wrote: > > I've just learned from Simon McVittie in #debian-devel that there is a way > > to > > cross-build packages using meson and gobject-introspection without this > > patch > > by configuring them with --cross-file. Here is an example from libportal: > > > > https://salsa.debian.org/debian/libportal/-/commit/4efec56ef735eac25836acca867fe59812c3dc6d Meson comes with a tool that converts native and cross environments into cross files. It is run with `meson env2mfile`. It even has a mode to generate Debian package building scripts. Updating that to handle this case would probably be the smartest thing to do. Currently Debian package building does not use that by default. We planned to make it use that, but it had some bugs and it had to be taken out for Debian release freeze.
Bug#1060838: meson: should use the host gobject-introspection-1.0.pc for looking up g-ir-scanner
On Wed, 21 Feb 2024 at 10:47:33 +0100, Johannes Schauer Marin Rodrigues wrote: > I've just learned from Simon McVittie in #debian-devel that there is a way to > cross-build packages using meson and gobject-introspection without this patch > by configuring them with --cross-file. Here is an example from libportal: > > https://salsa.debian.org/debian/libportal/-/commit/4efec56ef735eac25836acca867fe59812c3dc6d I did mention this cross-file in an earlier message to this same bug. This and other non-trivial gobject-introspection packaging topics are documented in: file:///usr/share/doc/gobject-introspection/README.Debian.gz (which I hope to keep up to date with best-practice as it changes). smcv
Bug#1060838: meson: should use the host gobject-introspection-1.0.pc for looking up g-ir-scanner
Hi, Quoting Helmut Grohne (2024-01-15 13:04:27) > Simon worked a lot on making gobject-introspection workable with cross > compilation and this work has hit unstable now. While this seems to work > fine with autotools, it tends to fail with meson. I've just learned from Simon McVittie in #debian-devel that there is a way to cross-build packages using meson and gobject-introspection without this patch by configuring them with --cross-file. Here is an example from libportal: https://salsa.debian.org/debian/libportal/-/commit/4efec56ef735eac25836acca867fe59812c3dc6d I successfully used the same technique for gst-plugins-bad1.0 and gst-plugins-good1.0. I'm recording this here so this technique doesn't get lost and in case others want to have their packages cross-built before this or a similar patch can get accepted. Thanks! cheers, josch signature.asc Description: signature
Bug#1060838: meson: should use the host gobject-introspection-1.0.pc for looking up g-ir-scanner
On Mon, 15 Jan 2024 at 14:32:44 +0100, Helmut Grohne wrote: > On Mon, Jan 15, 2024 at 12:38:54PM +, Simon McVittie wrote: > > in general the > > host g-ir-scanner will not even be executable during a cross-build > > I question this. g-ir-scanner is written in Python and it is fairly > unlikely that /usr/bin/python3 will end up being the host Python that > cannot be run. Hence using the g-ir-scanner from the host typically > should work (or not) the same way as the one from the build If it was pure Python, then this would be true, but it loads a private module written in C (/usr/lib/*/gobject-introspection/giscanner/_giscanner.cpython-311-*.so) and that module needs to match the architecture of the python3 interpreter. The result is that g-ir-scanner has two orthogonal architecture dependencies: 1. It needs to be loading a _giscanner module whose architecture matches python3: in practice this means adding /usr/lib/${DEB_BUILD_MULTIARCH}/gobject-introspection to sys.path 2. Meanwhile it also needs to be using a $CC, $PKG_CONFIG, and search paths that are appropriate for ${DEB_HOST_MULTIARCH}, and eventually run a temporary host-architecture dumper binary, possibly via qemu This makes it "the same shape" as gcc or ld, which are build-architecture executables that have host-architecture object code as their input/output. Unlike gcc, there is no built-in upstream way to build a cross-g-ir-scanner analogous to a cross-compiler, and there does not seem to be any interest in adding that upstream, which is why I had to do this as a Debian-specific change. However, reasoning similar to what's quoted above *does* work for tools like gdbus-codegen, which *is* pure Python, avoiding the first of those architecture dependencies. (As it happens, gdbus-codegen also operates at the level of source code rather than binaries, so it avoids the second architecture dependency too.) smcv
Bug#1060838: meson: should use the host gobject-introspection-1.0.pc for looking up g-ir-scanner
On Mon, Jan 15, 2024 at 12:38:54PM +, Simon McVittie wrote: > On Mon, 15 Jan 2024 at 13:04:27 +0100, Helmut Grohne wrote: > > Among other things, Simon introduced new programs > > /usr/bin/-g-ir-scanner. When cross building, these should be > > used and they'll automatically employ qemu-user when needed. He also > > updated gobject-introspection-1.0.pc to have g_ir_scanner point at this > > rather than plain /usr/bin/g-ir-scanner. Unfortunately, meson looks up > > the gobject-introspection-1.0 dependency with native set to true. Hence, > > it does not pick up the host triplet but the build triplet here and that > > doesn't go well. > > I believe this was done intentionally in Meson, because in general the > host g-ir-scanner will not even be executable during a cross-build. The > fact that we have made it work is Debian-specific. If I understand > upstream's position correctly, the way to force this sort of thing is > via cross-files. I question this. g-ir-scanner is written in Python and it is fairly unlikely that /usr/bin/python3 will end up being the host Python that cannot be run. Hence using the g-ir-scanner from the host typically should work (or not) the same way as the one from the build - unless one has specifically changed it (as you've done for the benefit of Debian). Possibly, some builds are performed with no gobject-introspection-1.0.pc being installed for the host architecture. In such cases, we could fall back from host architecture to build architecture. I don't see a plausible failure scenario arising from using the g_ir_scanner variable from the host file when available though. Independently of whether it breaks stuff or not, the proposed patch may still go against upstream's position, yes. > Debian's gobject-introspection does provide a cross-file that does what we > want (--cross-file=${DEB_HOST_GNU_TYPE}-gobject-introspection.ini). See > src:graphene for an example of this being used. Ideally, this would > be used automatically for all cross-builds that involve g-i, to avoid > needing to make sourceful changes in all packages that run g-i, but > unfortunately I am not aware of any currently available way to ask an > appropriate layer (debhelper?) to add a category of non-core cross files > to all cross-builds. > > Perhaps a possible route for the future would be to teach debhelper's > meson build system to add a --cross-file to all cross-builds, automatically, > for each file matching /usr/share/meson/cross/${DEB_HOST_GNU_TYPE}-*.ini? > But I have not yet discussed that idea with the debhelper maintainers. Let me express an argument against this approach. Being hooked into debhelper means that things will just work in Debian package builds. However, Debian is also used as a development platform for building other things than packages. When you perform your build by running meson and ninja directly, you'll have to remember to also pass these crossfiles for every cross build performed on Debian. In order for this to benefit non-package builds, meson itself would have to automatically consume those crossfiles. I actually see this as a problem in the rust ecosystem currently. Debian package builds are somehow enhanced to dependencies from Debian packages rather than downloading stuff from cargo. Replicating this offline experience without building Debian packages is a non-trivial affair. For this reasons, I still prefer a solution at the meson-level (even if it is not the solution I am proposing here). Still, if everyone else thinks that debhelper is the better place, I'm going out of the way and can supply the relevant debhelper patch. > I think we are going to need a similar mechanism in future for at > least Vala's vapigen (which is the reason why e.g. libportal can't be > cross-compiled by making changes similar to the ones in graphene). > > > def find_tool(self, name: str, depname: str, varname: str, required: > > bool = True, > > - wanted: T.Optional[str] = None) -> > > T.Union['build.Executable', ExternalProgram, 'OverrideProgram']: > > + wanted: T.Optional[str] = None, native: bool = True) -> > > T.Union['build.Executable', ExternalProgram, 'OverrideProgram']: > > Is find_tool intended to be analogous to Autoconf AC_CHECK_TOOL, > which automatically checks for a cross-tool using the convention that a > cross-tool that works with binaries for the host architecture ${HOST} (but > runnable on the build architecture) is usually named ${HOST}-${tool_name}, > falling back to ${tool_name} only if ${HOST}-${tool_name} is not found? > > (In Debian terms, ${HOST} is ${DEB_HOST_GNU_TYPE}) > > Or is find_tool intended to be more like Autoconf AC_CHECK_PROG, which is > appropriate for tools that have no particular architecture dependency, like > documentation generators, but not appropriate for compilers, linkers and > other tools that might have different behaviour and outputs per-architecture? Thanks for asking the question I also
Bug#1060838: meson: should use the host gobject-introspection-1.0.pc for looking up g-ir-scanner
On Mon, 15 Jan 2024 at 13:04:27 +0100, Helmut Grohne wrote: > Among other things, Simon introduced new programs > /usr/bin/-g-ir-scanner. When cross building, these should be > used and they'll automatically employ qemu-user when needed. He also > updated gobject-introspection-1.0.pc to have g_ir_scanner point at this > rather than plain /usr/bin/g-ir-scanner. Unfortunately, meson looks up > the gobject-introspection-1.0 dependency with native set to true. Hence, > it does not pick up the host triplet but the build triplet here and that > doesn't go well. I believe this was done intentionally in Meson, because in general the host g-ir-scanner will not even be executable during a cross-build. The fact that we have made it work is Debian-specific. If I understand upstream's position correctly, the way to force this sort of thing is via cross-files. Debian's gobject-introspection does provide a cross-file that does what we want (--cross-file=${DEB_HOST_GNU_TYPE}-gobject-introspection.ini). See src:graphene for an example of this being used. Ideally, this would be used automatically for all cross-builds that involve g-i, to avoid needing to make sourceful changes in all packages that run g-i, but unfortunately I am not aware of any currently available way to ask an appropriate layer (debhelper?) to add a category of non-core cross files to all cross-builds. Perhaps a possible route for the future would be to teach debhelper's meson build system to add a --cross-file to all cross-builds, automatically, for each file matching /usr/share/meson/cross/${DEB_HOST_GNU_TYPE}-*.ini? But I have not yet discussed that idea with the debhelper maintainers. I think we are going to need a similar mechanism in future for at least Vala's vapigen (which is the reason why e.g. libportal can't be cross-compiled by making changes similar to the ones in graphene). > def find_tool(self, name: str, depname: str, varname: str, required: > bool = True, > - wanted: T.Optional[str] = None) -> > T.Union['build.Executable', ExternalProgram, 'OverrideProgram']: > + wanted: T.Optional[str] = None, native: bool = True) -> > T.Union['build.Executable', ExternalProgram, 'OverrideProgram']: Is find_tool intended to be analogous to Autoconf AC_CHECK_TOOL, which automatically checks for a cross-tool using the convention that a cross-tool that works with binaries for the host architecture ${HOST} (but runnable on the build architecture) is usually named ${HOST}-${tool_name}, falling back to ${tool_name} only if ${HOST}-${tool_name} is not found? (In Debian terms, ${HOST} is ${DEB_HOST_GNU_TYPE}) Or is find_tool intended to be more like Autoconf AC_CHECK_PROG, which is appropriate for tools that have no particular architecture dependency, like documentation generators, but not appropriate for compilers, linkers and other tools that might have different behaviour and outputs per-architecture? smcv
Bug#1060838: meson: should use the host gobject-introspection-1.0.pc for looking up g-ir-scanner
Package: meson Version: 1.3.1-1 Severity: important Tags: patch upstream X-Debbugs-Cc: debian-cr...@lists.debian.org, s...@debian.org User: debian-cr...@lists.debian.org Usertags: ftcbfs Control: affects -1 + src:harfbuzz Hi, Simon worked a lot on making gobject-introspection workable with cross compilation and this work has hit unstable now. While this seems to work fine with autotools, it tends to fail with meson. Among other things, Simon introduced new programs /usr/bin/-g-ir-scanner. When cross building, these should be used and they'll automatically employ qemu-user when needed. He also updated gobject-introspection-1.0.pc to have g_ir_scanner point at this rather than plain /usr/bin/g-ir-scanner. Unfortunately, meson looks up the gobject-introspection-1.0 dependency with native set to true. Hence, it does not pick up the host triplet but the build triplet here and that doesn't go well. We need to change the native argument to false here. I'm attaching a patch demonstrating the change and it makes building e.g. harfbuzz work. I expect this change to fix many packages hence raising the severity to important. I'm not sure whether this is the desired way to extend meson nor whether this has consequences for downstreams other than Debian. How can we take care of these matters? Helmut --- meson-1.3.1.orig/mesonbuild/modules/__init__.py +++ meson-1.3.1/mesonbuild/modules/__init__.py @@ -96,7 +96,7 @@ wanted=wanted, silent=silent, for_machine=for_machine) def find_tool(self, name: str, depname: str, varname: str, required: bool = True, - wanted: T.Optional[str] = None) -> T.Union['build.Executable', ExternalProgram, 'OverrideProgram']: + wanted: T.Optional[str] = None, native: bool = True) -> T.Union['build.Executable', ExternalProgram, 'OverrideProgram']: # Look in overrides in case it's built as subproject progobj = self._interpreter.program_from_overrides([name], []) if progobj is not None: @@ -108,7 +108,7 @@ return ExternalProgram.from_entry(name, prog_list) # Check if pkgconfig has a variable -dep = self.dependency(depname, native=True, required=False, wanted=wanted) +dep = self.dependency(depname, native=native, required=False, wanted=wanted) if dep.found() and dep.type_name == 'pkgconfig': value = dep.get_variable(pkgconfig=varname) if value: --- meson-1.3.1.orig/mesonbuild/modules/gnome.py +++ meson-1.3.1/mesonbuild/modules/gnome.py @@ -328,7 +328,7 @@ } depname = tool_map[tool] varname = tool.replace('-', '_') -return state.find_tool(tool, depname, varname) +return state.find_tool(tool, depname, varname, native=depname != "gobject-introspection-1.0") @typed_kwargs( 'gnome.post_install',