Bug#859177: meson is unuseable for package cross compilation

[Jussi Pakkanen]

On Sun, Jul 30, 2017 at 8:20 AM, Helmut Grohne  wrote:

> When debcrossgen.py is shipped in a "known" location (e.g. $PATH), I
> can simply run it myself, modify its output, and pass "--cross-file
> myfile" to dh_auto_configure. The later --cross-file takes precedence as
> far as I understand.

That is an implementation detail of the way Python argument parser
works. It might or might not work in the future. Regardless of that it
is clearer (in my opinion at least) to have only one of any kind of
argument so it would be better to have a workflow that specifies only
one cross file.

Bug#859177: meson is unuseable for package cross compilation

[Helmut Grohne]

On Sun, Jul 30, 2017 at 02:07:50AM +0200, Michael Biebl wrote:
> You mean as part of the Debian package build process? How would it make
> a difference if debhelper calls an external binary or generate the file
> directly? Can you please elaborate

When debcrossgen.py is shipped in a "known" location (e.g. $PATH), I
can simply run it myself, modify its output, and pass "--cross-file
myfile" to dh_auto_configure. The later --cross-file takes precedence as
far as I understand.

Helmut

Bug#859177: meson is unuseable for package cross compilation

[Michael Biebl]

Am 29.07.2017 um 23:29 schrieb Helmut Grohne:
> On Sat, Jul 29, 2017 at 09:12:56AM +0200, Michael Biebl wrote:

>> dependency on Python. But the  debcrossgen.py code looks straightforward
>> enough to be ported to perl.
> 
> If it ends up being shipped in debhelper, I have a few questions. What
> should I do if the cross file mostly is correct, but I need to tweak one
> line (i.e. how to override it)?

You mean as part of the Debian package build process? How would it make
a difference if debhelper calls an external binary or generate the file
directly? Can you please elaborate




-- 
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?



signature.asc
Description: OpenPGP digital signature

Bug#859177: meson is unuseable for package cross compilation

[Helmut Grohne]

On Sat, Jul 29, 2017 at 09:12:56AM +0200, Michael Biebl wrote:
> Am 28.07.2017 um 22:06 schrieb Helmut Grohne:
>
> >  2. The debcrossgen.py script needs some place to live. When debhelper
> > and meson are installed together, it needs to be in the filesystem.
> > An easy approach to do so is to put it into the meson binary
> > package. We should probably spend a little discussing bike shedding
> > questions such as:
> >  a. What path/filename to use?
> >  b. What options/arguments should it take?
> >  3. debhelper needs to call debcrossgen.py during cross compilation and
> > add the --cross-file option. This one is straight forward to
> > implement once the questions from 2. are answered.
>
> My gut feeling is, that this functionality should be shipped in
> debhelper directly, specifically in the meson build system class [1].
> debhelper is written in perl, and I don't think we want to add a
> dependency on Python. But the  debcrossgen.py code looks straightforward
> enough to be ported to perl.

If it ends up being shipped in debhelper, I have a few questions. What
should I do if the cross file mostly is correct, but I need to tweak one
line (i.e. how to override it)? What should I do if I want to cross
build for a Debian-based system, but don't want to build a Debian
package, just a binary? (Do I have to write the cross file manually?)

> That said, if Jussi thinks that this script might be useful for other
> distros / use cases, then shipping it upstream in the meson package
> might be an option as well.

Indeed, the tool could be extended to be useful for other distros. There
is a de facto standard for describing architectures used across a number
of distributions and that is the GNU triplet (i.e. DEB_HOST_GNU_TYPE).
If the tool were deriving all of the values from the GNU triplet rather
than checking all dpkg-architecture variables, it could become useful
for others. At the same time, it would essentially duplicate autotools'
config.guess. Choose either simple or generic, but not both.

On Sat, Jul 29, 2017 at 09:19:22AM +0200, Michael Biebl wrote:
> While generating a (temporary) cross.txt file is not a huge deal, being
> able to directly pass the necessary options to meson via command line
> options would be a tad nicer.

That's what we do for CMake indeed, but it is not without problems
there. Some build systems behave differently when a CMAKE_TOOLCHAIN_FILE
is given and assume a non-cross build when there is none. That pitfall
should be avoided.

Helmut

Bug#859177: meson is unuseable for package cross compilation

[Jussi Pakkanen]

On Sat, Jul 29, 2017 at 10:19 AM, Michael Biebl  wrote:

> Jussi, would this be hard to implement? Is there a good reason why
> cross-compilation uses a separate file?

There are many reasons. The fils contains a lot of options and passing
all those in command line arguments would make for very bloated
command lines. Since the data in cross files rarely changes, having it
a standalone file makes it easy to share between projects, commit them
in revision control, and so on.

Bug#859177: meson is unuseable for package cross compilation

[Michael Biebl]

Am 29.07.2017 um 09:12 schrieb Michael Biebl:

> My gut feeling is, that this functionality should be shipped in
> debhelper directly, specifically in the meson build system class [1].
> debhelper is written in perl, and I don't think we want to add a
> dependency on Python. But the  debcrossgen.py code looks straightforward
> enough to be ported to perl.

While generating a (temporary) cross.txt file is not a huge deal, being
able to directly pass the necessary options to meson via command line
options would be a tad nicer.

Jussi, would this be hard to implement? Is there a good reason why
cross-compilation uses a separate file?


-- 
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?



signature.asc
Description: OpenPGP digital signature

Bug#859177: meson is unuseable for package cross compilation

[Michael Biebl]

Am 28.07.2017 um 22:06 schrieb Helmut Grohne:

>  2. The debcrossgen.py script needs some place to live. When debhelper
> and meson are installed together, it needs to be in the filesystem.
> An easy approach to do so is to put it into the meson binary
> package. We should probably spend a little discussing bike shedding
> questions such as:
>  a. What path/filename to use?
>  b. What options/arguments should it take?
>  3. debhelper needs to call debcrossgen.py during cross compilation and
> add the --cross-file option. This one is straight forward to
> implement once the questions from 2. are answered.

My gut feeling is, that this functionality should be shipped in
debhelper directly, specifically in the meson build system class [1].
debhelper is written in perl, and I don't think we want to add a
dependency on Python. But the  debcrossgen.py code looks straightforward
enough to be ported to perl.

That said, if Jussi thinks that this script might be useful for other
distros / use cases, then shipping it upstream in the meson package
might be an option as well.

Jussi, what would you prefer?

Michael


[1]
https://anonscm.debian.org/git/debhelper/debhelper.git/tree/Debian/Debhelper/Buildsystem/meson.pm


-- 
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?



signature.asc
Description: OpenPGP digital signature

Bug#859177: meson is unuseable for package cross compilation

[Helmut Grohne]

Hi,

I took a while to reply for a number of reasons:
 * There wasn't anything left to disagree with.
 * Testing debcrossgen.py was difficult due to #859173.

In the mean time:
 * ninja-build is marked M-A:foreign (i.e. #859173 is fixed).
 * systemd can be built with meson.
 * The Debian systemd package is built with meson.

=> We have a test case!

So I went ahead and tried making it cross compile.

On Sun, Apr 02, 2017 at 07:50:33PM +0300, Jussi Pakkanen wrote:
> Attached is a simple script that generates a cross file that works for
> me when using a simple project. Use it like this:
> 
> ./debcrossgen.py armhf generated_cross.txt
> 
> And then pass --libdir=lib/archdir_here --cross-file
> generated_cross.txt to Meson to use it.
> 
> It only takes care of host, not target. Consider it an MVP. :)

That looks like a very good start.

I dropped that script into systemd's Debian packaging with slight
modifications and made systemd execute it during cross compilation. I
annotated the Build-Depends: meson with :native (which is accepted by
sbuild) and cheated dpkg-buildpackage with -d. To my surprise, systemd
just cross built successfully on the first attempt. The patch is
attached, but it is not meant to be applied to systemd. It just serves
as a proof of feasibility.

Essentially there are three issues that need fixing to make this work
properly:
 1. Dependencies on meson need to become cross-satisfiable. We discussed
that already and essentially concluded that meson should follow
cmake and become M-A:foreign. Any objections?
 2. The debcrossgen.py script needs some place to live. When debhelper
and meson are installed together, it needs to be in the filesystem.
An easy approach to do so is to put it into the meson binary
package. We should probably spend a little discussing bike shedding
questions such as:
 a. What path/filename to use?
 b. What options/arguments should it take?
 3. debhelper needs to call debcrossgen.py during cross compilation and
add the --cross-file option. This one is straight forward to
implement once the questions from 2. are answered.

Would it make sense to split this bug report into three?

So let's focus on debcrossgen.py. I'm not sure whether it needs to live
on $PATH. Due to its use of dpkg-architecture the script is probably
useless on non-Debian systems. I think passing the architecture should
be optional or disabled: dpkg-architecture looks up the host
architecture from the environment. When dropping the architecture from
the command line, "./debcrossgen.py armhf output.txt" becomes
"dpkg-architecture -aarmhf -c ./debcrossgen.py output" and while
building a package, those environment variables are already set up
correctly. Optimizing for the common case means not requesting the
architecture name.

The output of debcrossgen.py still is subtly wrong for a number of
architectures including armhf, musl-linux-mips, kfreebsd-amd64,
hurd-i386 and a few more. I think that's ok. We should start shipping it
and fix the output later. That's what we did for cmake (in debhelper)
and it worked well.

Helmut
diff --minimal -Nru systemd-234/debian/changelog systemd-234/debian/changelog
--- systemd-234/debian/changelog2017-07-20 15:13:42.0 +0200
+++ systemd-234/debian/changelog2017-07-27 21:50:40.0 +0200
@@ -1,3 +1,10 @@
+systemd (234-2.1) UNRELEASED; urgency=medium
+
+  * Non-maintainer upload.
+  * 
+
+ -- Helmut Grohne   Thu, 27 Jul 2017 21:50:40 +0200
+
 systemd (234-2) unstable; urgency=medium
 
   [ Martin Pitt ]
diff --minimal -Nru systemd-234/debian/control systemd-234/debian/control
--- systemd-234/debian/control  2017-07-20 15:13:42.0 +0200
+++ systemd-234/debian/control  2017-07-27 21:50:39.0 +0200
@@ -16,7 +16,7 @@
docbook-xsl,
docbook-xml,
m4,
-   meson (>= 0.40.0),
+   meson:native (>= 0.40.0),
intltool,
gperf,
gnu-efi [amd64 i386 arm64],
diff --minimal -Nru systemd-234/debian/debcrossgen.py 
systemd-234/debian/debcrossgen.py
--- systemd-234/debian/debcrossgen.py   1970-01-01 01:00:00.0 +0100
+++ systemd-234/debian/debcrossgen.py   2017-07-27 21:50:40.0 +0200
@@ -0,0 +1,48 @@
+#!/usr/bin/env python3
+
+# Copyright 2017 Jussi Pakkanen
+
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+
+# http://www.apache.org/licenses/LICENSE-2.0
+
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import sys, os, subprocess
+
+def run():
+output = 

Bug#859177: meson is unuseable for package cross compilation

[Jussi Pakkanen]

On Sat, Apr 1, 2017 at 7:45 AM, Helmut Grohne  wrote:

>> That being said generating them live is probably a good approach, it
>> seems to be mostly a question of mangling the output of
>> dpkg-architecture.
>
> Yes. That's also essentially how the command line parameters for CMake
> are constructed[1].

Attached is a simple script that generates a cross file that works for
me when using a simple project. Use it like this:

./debcrossgen.py armhf generated_cross.txt

And then pass --libdir=lib/archdir_here --cross-file
generated_cross.txt to Meson to use it.

It only takes care of host, not target. Consider it an MVP. :)
#!/usr/bin/env python3

# Copyright 2017 Jussi Pakkanen

# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at

# http://www.apache.org/licenses/LICENSE-2.0

# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

import sys, os, subprocess

def run(arch, ofilename):
output = subprocess.check_output(['dpkg-architecture', '-a' + arch], universal_newlines=True)
data = {}
for line in output.split('\n'):
line = line.strip()
if line == '':
continue
k, v = line.split('=', 1)
data[k] = v
host_arch = data['DEB_HOST_GNU_TYPE']
host_os = data['DEB_HOST_ARCH_OS']
host_cpu_family = data['DEB_HOST_GNU_CPU']
host_cpu = data['DEB_HOST_ARCH'] # Not really correct, should be arm7hlf etc but it is not exposed.
host_endian = data['DEB_HOST_ARCH_ENDIAN']
ofile = open(ofilename, 'w')
ofile.write('[binaries]\n')
ofile.write("c = '/usr/bin/%s-gcc-6'\n" % host_arch)
ofile.write("cpp = '/usr/bin/%s-g++-6'\n" % host_arch)
ofile.write("ar = '/usr/bin/%s-ar'\n" % host_arch)
ofile.write("strip = '/usr/bin/%s-strip'\n" % host_arch)
ofile.write("pkgconfig = '/usr/bin/%s-pkg-config'\n" % host_arch)
ofile.write('\n[properties]\n')
ofile.write('\n[host_machine]\n')
ofile.write("system = '%s'\n" % host_os)
ofile.write("cpu_family = '%s'\n" % host_cpu_family)
ofile.write("cpu = '%s'\n" % host_cpu)
ofile.write("endian = '%s'\n" % host_endian)
ofile.close()

if __name__ == '__main__':
if len(sys.argv) != 3:
print(sys.argv[0], 'host-arch outputfile')
sys.exit(1)
arch = sys.argv[1]
outfile = sys.argv[2]
run(arch, outfile)
print('Remember to add the proper --libdir arg to Meson invocation.')

Bug#859177: meson is unuseable for package cross compilation

[Helmut Grohne]

On Sat, Apr 01, 2017 at 02:01:07AM +0300, Jussi Pakkanen wrote:
> On Fri, Mar 31, 2017 at 4:31 PM, Helmut Grohne  wrote:
> 
> > The `makefile` buildsystem in `debhelper` even just sets 
> > `CC=${DEB_HOST_GNU_TYPE}-gcc`. The convention is: prefix host tools with 
> > the host's GNU triplet.
> 
> This does not work. For Meson, CC always means "the native compiler".
> This is just a matter of unsetting that var (or making it point to
> /usr/bin/cc, whichever is simpler).

It was just meant as an example for how the compiler name is being
constructed. The way of telling meson can certainly be different.

> > I agree that shipping this as a Debian-ism may prove easier for
> > development of the thing. As you are the maintainer and agree with that
> > idea, it sounds like a plan. Still, I would like to avoid using any
> > Debianisms in the implementation (as CMake still does in some parts) to
> > ease upstreaming down the road.
> 
> AFAICR CMake has a similar concept of a toolchain file for cross
> compilations. How are they dealt with in Debian?

CMake does not mandate the use of a toolchain file. Instead it mixes
the toolchain definitions into its variable scope and thus they can be
changed like any other user configurable option with a suitable
`-Dfoo=bar` flag. Then `debhelper` sets[1] lots of them.

> In Meson we avoid making absolutely everything configurable in a
> gazillion different ways. It just leads to frustration. We prefer
> explicit data in (preferably only one) place and for cross compilation
> that is the cross file. Having one plain text file with all the
> information makes e.g. debugging a lot simpler.

That makes sense to me. Indeed that way simplifies approaching meson.

> Those numbers are a bit off. Information about the build machine is
> not stored in the cross file, it is detected at runtime, identically
> to how native building would work. Thus cross compiling for 36
> architectures would require 36 files or 72 if accounting for cases
> where target is specified but host is not. You'd only need to go to
> 36*36 = 1296 files if you want to support cross compiling cross
> compilers on all possible combinations.

The difficulty here is deciding in advance which combinations are
relevant. However you decide, tomorrow a new arch will come along and
you'll have to add it posing a constant stream of maintenance. I've seen
that with `libgpg-error`, `guile-2.0`, `jemalloc`, etc. Trust me: It's a
pain (at least for me having to report all those bugs). If that 36 were
a mostly static number, that'd draw a different picture.

> That being said generating them live is probably a good approach, it
> seems to be mostly a question of mangling the output of
> dpkg-architecture.

Yes. That's also essentially how the command line parameters for CMake
are constructed[1].

> That is old code that is not used much any more. Nowadays we don't do
> library lookups based on file paths but instead by asking the compiler
> (i.e. we know library 'foo' is available if compilation with the
> desired compiler when using '-lfoo' passes.
> 
> Still, that should probably be fixed at some point.

Glad to hear that. As a first step, maybe update the docstrings telling
that these shouldn't be used pointing to the alternatives?

I have the feeling that meson gets quite a few things "right" (from a
cross build maintainer's view). Too bad we're late for stretch. I was
almost considering switching some CMake projects to meson due to the
much better handling of build/host/target.

Helmut

[1] 
http://sources.debian.net/src/debhelper/10.2.5/Debian/Debhelper/Buildsystem/cmake.pm/#L59

Bug#859177: meson is unuseable for package cross compilation

[Jussi Pakkanen]

On Fri, Mar 31, 2017 at 4:31 PM, Helmut Grohne  wrote:

> The `makefile` buildsystem in `debhelper` even just sets 
> `CC=${DEB_HOST_GNU_TYPE}-gcc`. The convention is: prefix host tools with the 
> host's GNU triplet.

This does not work. For Meson, CC always means "the native compiler".
This is just a matter of unsetting that var (or making it point to
/usr/bin/cc, whichever is simpler).

> I agree that shipping this as a Debian-ism may prove easier for
> development of the thing. As you are the maintainer and agree with that
> idea, it sounds like a plan. Still, I would like to avoid using any
> Debianisms in the implementation (as CMake still does in some parts) to
> ease upstreaming down the road.

AFAICR CMake has a similar concept of a toolchain file for cross
compilations. How are they dealt with in Debian?

> The more radical approach would be adding a `--host` or `--host-arch`
> option to `meson` itself. It could be setting different defaults and
> still allow passing a `--cross-file` to override.

In Meson we avoid making absolutely everything configurable in a
gazillion different ways. It just leads to frustration. We prefer
explicit data in (preferably only one) place and for cross compilation
that is the cross file. Having one plain text file with all the
information makes e.g. debugging a lot simpler.

>> There is, however, an alternative to this. There is only one cross
>> file per architecture, so for Debian a few dozen. Those could be
>> manually generated and provided as part of dh_meson package or
>> somesuch. Then getting a cross compilation going would be just a case
>> of adding this to the configuration line:
>>
>> --cross-file /usr/share/something/triplet-filename.txt
>
> The idea is correct in principle, but it suffers from three flaws.
> Beyond build and host architectures, meson appears to have limited
> support for target (as you mentioned below). Adding the target into the
> mix means squaring the number of files. The second flaw is that some
> aspects of the build architecture may vary. If that turns out to happen,
> the number of files becomes the number of architectures to the power of
> three. I'm working on 36 architectures, so 46656 files. dpkg knows about
> 523 architectures, i.e. 143055667 files.

Those numbers are a bit off. Information about the build machine is
not stored in the cross file, it is detected at runtime, identically
to how native building would work. Thus cross compiling for 36
architectures would require 36 files or 72 if accounting for cases
where target is specified but host is not. You'd only need to go to
36*36 = 1296 files if you want to support cross compiling cross
compilers on all possible combinations.

That being said generating them live is probably a good approach, it
seems to be mostly a question of mangling the output of
dpkg-architecture.

> Briefly looking into `mesonlib.default_libdir`, it seems that `meson`
> already hooks into Debian-specifics. The hack in
> `mesonlib.get_library_dirs` looks even worse as it will be wrong on a
> pile of architectures and for cross compilation. I wonder why
> `dpkg-architecture` is used in `default_libdir`, but not
> `get_library_dirs`. That looks inconsistent to me. :-(

That is old code that is not used much any more. Nowadays we don't do
library lookups based on file paths but instead by asking the compiler
(i.e. we know library 'foo' is available if compilation with the
desired compiler when using '-lfoo' passes.

Still, that should probably be fixed at some point.

Bug#859177: meson is unuseable for package cross compilation

[Helmut Grohne]

On Fri, Mar 31, 2017 at 02:41:55PM +0300, Jussi Pakkanen wrote:
> Meson requires access to the "native" and "cross" compilers at the
> same time. This is because it natively supports the use case where you
> compile a program that is then used to generate source code that is
> built. Examples include idl compilers and the like. I'm not familiar
> enough with the Debian cross compilation environment to know if those
> are available.

The standard way to get the build architecture compiler (what you call
native) is to simple use the plain compiler (e.g. `gcc`) without any
fuss.

The standard way to get the host architecture compiler (what you call
cross) is the plain compiler name prefixed with the GNU triplet. Let me
give an example here. If you cross build for `armhf` (Debian
architecture), `dpkg-architecture -aarmhf` tells us that the GNU triplet
is `arm-linux-gnueabihf`. Thus the actual compiler is called
`arm-linux-gnueabihf-gcc`. The `makefile` buildsystem in `debhelper`
even just sets `CC=${DEB_HOST_GNU_TYPE}-gcc`. The convention is: prefix
host tools with the host's GNU triplet.

This approach doesn't just work for `gcc`, it works for all of gcc's
frontends, binutils, pkg-config, and even python-config. It also tends
to work on other distributions (once you know the GNU triplet).

> Something like this could be done and possibly even provided upstream.
> However I'd like it to first go to Debian for a while. If it lives
> upstream it would need to support many distros (at least Debian +
> Fedora, preferably also Arch and the like). Getting it running on a
> single distro is a lot simpler.

I agree that shipping this as a Debian-ism may prove easier for
development of the thing. As you are the maintainer and agree with that
idea, it sounds like a plan. Still, I would like to avoid using any
Debianisms in the implementation (as CMake still does in some parts) to
ease upstreaming down the road.

The more radical approach would be adding a `--host` or `--host-arch`
option to `meson` itself. It could be setting different defaults and
still allow passing a `--cross-file` to override.

> There is, however, an alternative to this. There is only one cross
> file per architecture, so for Debian a few dozen. Those could be
> manually generated and provided as part of dh_meson package or
> somesuch. Then getting a cross compilation going would be just a case
> of adding this to the configuration line:
> 
> --cross-file /usr/share/something/triplet-filename.txt

The idea is correct in principle, but it suffers from three flaws.
Beyond build and host architectures, meson appears to have limited
support for target (as you mentioned below). Adding the target into the
mix means squaring the number of files. The second flaw is that some
aspects of the build architecture may vary. If that turns out to happen,
the number of files becomes the number of architectures to the power of
three. I'm working on 36 architectures, so 46656 files. dpkg knows about
523 architectures, i.e. 143055667 files. Furthermore, we'll sometimes
have to "fix" these `--cross-file`s for individual source packages.
Thus the `cross-meson` utility should be able to receive a
`--cross-file` overlay to be able to override any setting a maintainer
wishes to change. Otherwise, they'll just opt out of using `cross-meson`
and get things wrong. Keeping the flexibility seems important to me.

For these reasons, I believe that just storing them statically is
infeasible.

> This is especially useful if there are platforms that require tweaks
> to the cross files that the generator would create. This gets a bit
> more complicated if compiling cross compilers or binutils, which would
> require having a different cross file where host is different from
> target. At the time of writing there are no compilers building with
> Meson that I know of so this is not an acute issue.

Before too long, I expect `meson` to be picked up for compilers. We
should be prepared for that case.

Briefly looking into `mesonlib.default_libdir`, it seems that `meson`
already hooks into Debian-specifics. The hack in
`mesonlib.get_library_dirs` looks even worse as it will be wrong on a
pile of architectures and for cross compilation. I wonder why
`dpkg-architecture` is used in `default_libdir`, but not
`get_library_dirs`. That looks inconsistent to me. :-(

Does any reader of `debian-cr...@lists.debian.org` object to just
marking `meson` `Multi-Arch: foreign` on the grounds that even if it may
be wrong, any other value makes even less sense?

Helmut

Bug#859177: meson is unuseable for package cross compilation

[Jussi Pakkanen]

On Fri, Mar 31, 2017 at 10:37 AM, Helmut Grohne  wrote:

> When running `meson` on a project without supplying a `--cross-file`,
> `meson` will pick default system compilers. Those compilers will produce
> architecture-dependent output, which `meson` inherits. Thus `meson`'s
> interfaces certainly are not architecture-independent (and therefore
> suitable for `Multi-Arch: foreign`) in the obvious way. That said, the
> same problem holds for `make` and `cmake`. `make` was originally marked
> `Multi-Arch: foreign` until Jakub Wilk noticed that its handling of
> filesystem paths is architecture-dependent and it is now `Multi-Arch:
> allowed` (and considered architecture independent through
> build-essential unless one explicitly `Build-Depends: make`). `cmake` is
> marked `Multi-Arch: foreign`. So we need to define a "reasonable use" of
> meson and evaluate whether such use is indeed architecture-independent.

The simplest approach is to follow what CMake does because what we do
is roughly similar. However there is one extra kink here.

Meson requires access to the "native" and "cross" compilers at the
same time. This is because it natively supports the use case where you
compile a program that is then used to generate source code that is
built. Examples include idl compilers and the like. I'm not familiar
enough with the Debian cross compilation environment to know if those
are available.

> Thus `debhelper` would simply call `cross-meson` with the appropriate
> GNU triplets (that it can easily derive using `dpkg-architecture`) and
> skip any cross compilation specific complexity for `meson`. Such a
> script would be useful beyond Debian as GNU triplets are a very common
> thing during cross compilation. Rather than having each and every Linux
> distribution produce its own `--cross-file` generator, why not have this
> upstream? Of course I am not proposing to abolish the more flexible
> `--cross-file` approach: It does server actual use cases beyond
> well-maintained Linux distributions. But maybe we can have a wrapper to
> handle the common case?

Something like this could be done and possibly even provided upstream.
However I'd like it to first go to Debian for a while. If it lives
upstream it would need to support many distros (at least Debian +
Fedora, preferably also Arch and the like). Getting it running on a
single distro is a lot simpler.

There is, however, an alternative to this. There is only one cross
file per architecture, so for Debian a few dozen. Those could be
manually generated and provided as part of dh_meson package or
somesuch. Then getting a cross compilation going would be just a case
of adding this to the configuration line:

--cross-file /usr/share/something/triplet-filename.txt

This is especially useful if there are platforms that require tweaks
to the cross files that the generator would create. This gets a bit
more complicated if compiling cross compilers or binutils, which would
require having a different cross file where host is different from
target. At the time of writing there are no compilers building with
Meson that I know of so this is not an acute issue.

Bug#859177: meson is unuseable for package cross compilation

[Helmut Grohne]

Package: meson
Severity: wishlist
Control: block -1 by 859173

Mchael Biebl (Cced) asked me to look into cross compilation with
`meson`.  I already filed a related bug #859173 for the underlying
`ninja-build`, but `meson` itself also poses problems for cross
compilation.

For starters, any package that `Build-Depends: meson` cannot satisfy its
cross build dependencies, because `meson` is `Architecture: all` and
implicitly `Multi-Arch: no`. Such packages can never satisfy cross build
dependencies. There are essentially two approaches for solving this
problem. One involves changing the `Architecture` (makes little sense
here) and the other involves adding `Multi-Arch: foreign`, but it is not
clear to me that the latter is correct.

When running `meson` on a project without supplying a `--cross-file`,
`meson` will pick default system compilers. Those compilers will produce
architecture-dependent output, which `meson` inherits. Thus `meson`'s
interfaces certainly are not architecture-independent (and therefore
suitable for `Multi-Arch: foreign`) in the obvious way. That said, the
same problem holds for `make` and `cmake`. `make` was originally marked
`Multi-Arch: foreign` until Jakub Wilk noticed that its handling of
filesystem paths is architecture-dependent and it is now `Multi-Arch:
allowed` (and considered architecture independent through
build-essential unless one explicitly `Build-Depends: make`). `cmake` is
marked `Multi-Arch: foreign`. So we need to define a "reasonable use" of
meson and evaluate whether such use is indeed architecture-independent.

To ease such use and to simplify adding a `meson` buildsystem to
`debhelper` (#795253), I propose that `meson` gains a supporting script.
For `autotools`, it is customary, that one specifies the build, host and
target architectures as GNU triplets at configure time. No further cross
configuration is necessary and that makes cross building
`autotools`-based projects essentially just work. Contrast that with
`meson`, where the user has to supply an elaborate `--cross-file`.
`meson`'s approach adds a lot of flexibility over `autotools`' ones at
the cost of complexity.  Thus I propose that we (or even upstream) adds
a shell script which takes three additional parameters (build, host and
target architectures in GNU triplet format) and produces a suitable
`--cross-file` before invoking `meson`. We might call it `cross-meson`.
Thus `debhelper` would simply call `cross-meson` with the appropriate
GNU triplets (that it can easily derive using `dpkg-architecture`) and
skip any cross compilation specific complexity for `meson`. Such a
script would be useful beyond Debian as GNU triplets are a very common
thing during cross compilation. Rather than having each and every Linux
distribution produce its own `--cross-file` generator, why not have this
upstream? Of course I am not proposing to abolish the more flexible
`--cross-file` approach: It does server actual use cases beyond
well-maintained Linux distributions. But maybe we can have a wrapper to
handle the common case?

No code for the `cross-meson` idea has materialized at the time of this
writing. Normally, I'd only file such a bug with a patch, but Michael
asked me to get the discussion going as soon as possible. Discussion
will be necessary as the `meson` maintainers will likely know little
about cross compilation and `debian-cr...@lists.debian.org` members such
as myself have little knowledge of `meson`. Only joining forces will
lead to success (as has happened e.g. in a fruitful discussion with
TeXlive maintainer Norbert Preining).

Hope this helps

Helmut