Bug#771070: requirements for cross toolchain packages in the distribution
+++ Ben Longbons [2014-12-18 12:23 -0800]:
On Thu, Dec 18, 2014 at 8:36 AM, Wookey woo...@wookware.org wrote:
MA-built vs in-arch
---
I guess an interesting question is 'what does the cross-compiler
actually _use_ the foreign arch libc for'? Does it need its own
independent copy? What happens when the compiler libc-$arch-cross and
the system libc:$arch get out of sync. Does it matter?
The thought of this gives me nightmares. Glibc is very good at
backwards-compatibility as far as packages go, but does not even
attempt forwards-compatibility.
Indeed. This seems like a bad thing, but in fairness, emdebian and
ubuntu have been shipping cross-compilers where the versions do not
necessarily match for quite some time, and it hasn't caused obvious
practical problems.
So maybe this doesn't matter, but I'd really like to understand
exactly what's going on so we could see what might break, and thus
allocate an importance.
In practice I presume that all these libraries are dynamically linked
so in fact what matters is what versions are available at runtime, and
how tight the dependencies are.
If anything (I'm still confused if this means only libstdc++, or every
user program - but for those of us writing C++ I suppose it makes no
difference) gets built against libc-$arch-cross (= 2.999) but at
runtime there is only libc:$arch (= 2.998), then programs will almost
certainly fail to load because of missing symbol versions, and
possibly even fail to link.
In general, even if the compiler is built against libc-$arch-cross,
(and thus it is installed along with the toolchain) programs built
with that compiler should link against libc:$arch. I have seen cases
where they don't, and end up linking against the libc-$arch-cross copy
instead. I don't have a good handle on how common this is.
But clearly, if there isn't a libc-$arch-cross 'internal' copy present
then this problem can't arise. On the other hand, having the two
'flavours' of libc lets you install the toolchain 'within-arch', and
avoids uninstallability-due-to-multiarch-skew in unstable.
multilib vs mulitarch
-
Native compilers are not yet co-installable so have to use multilib to
target more than one ABI.
Can we please fix this? I'm tired of having to special-case my
buildbot scripts for arches only available through multilib (i586 and
gnux32 on an amd64 host; this also prevents ever running my buildbot
on anything other than amd64). For all other arches (native or
multiarch cross) the script is trivial.
This is the relevant page:
https://wiki.debian.org/CoinstallableToolchains
which has links to the bug at the bottom:666743. Helmut has the best
handle on the changes needed for this, and so far as I know Doko did
not object to the moving around of symlinks which makes this possible
(it was discussed at the bootstrap sprint
https://wiki.debian.org/Sprints/2014/BootstrapSprint/Results section 3.11).
Up to date patches would be good, to get this moving.
It does not work to just make my own ~/bin/i586-linux-gnu-gcc etc
scripts that call x86_64-linux-gnu-gcc -m32, because I also have to
worry about the libs being in the wrong place ({/usr,}/lib32 instead
of {/usr,}/lib/i586-linux-gnu, except that it's really
{/usr,}/lib/i386-linux-gnu ... why can't the numbers be consistent?
It's even i486-linux-gnu in places too ... meh, unimportant to this
discussion, except I suppose I'll need to special-case the library
locations anyway, or else lie any call my binaries packages i386 even
though they're really i586 ... it's not like everyone else doesn't
anyway).
Yeah that's all a pain. Blame the x86 people for using 3 different
triplets for the same ABI.
Multilibs do make much better sense for arches that are not in the
archive (x32/mipsn32) and are possibly the easier way to support
those, but even there, all the difficulty is in getting the right
libc:$arch or libc-$arch-cross packages. Once you have those you can
build per-arch toolchains or multilib toolchains.
The most correct solution to me seems to be a libs only archive even
for unsupported arches. This would be a huge win even for supported
arches, because 'apt-get update' with N architectures enabled is
really slow already.
Right. This is the logical outcome of using multiarch for library
dependencies. If you don't do this you have treat supported and
unsupported arches differently.
Conceptually it would be split into 3 areas that I can think of:
1. anything that ships architecture-specific binaries (not usable in
cross situations in general, but there are still useful packages like
wine32)
2. anything that ships architecture-specific libraries but not
binaries (useful for cross)
3. anything that ships no architecture-specific binaries or libraries
2 is Multi-Arch: same and 3 is Architecture: all, so we already have
an easy way to identify these sets of packages. That said, I can't
think of any
Bug#771070: requirements for cross toolchain packages in the distribution
[Most of this was written a few weeks ago when this bug was first filed, but I didn't post it then as I wanted to get better docs on the compiler builds in place at https://wiki.debian.org/ToolChain/Cross so that the substantive argument was easier to follow for those not previously involved. That could still do with more work, but it's time to actually reply, so sending. Some of this has since also been said in 766708 ] Matthias Klose d...@debian.org wrote: Recently some crippled cross toolchain packages were uploaded to Debian unstable, I do wish you'd stop calling these toolchains 'crippled' every time you refer to them. They are the only (so far), and thus 'best', cross-toolchains uploaded to debian itself and meet the requirements of most users. In fact there are only two people who have expressed any dissatisfaction with them so far. You and Dmitry. Quite a few other people have been very pleased to see them. I do not claim that they are perfect, and expect there to be changes for Stretch, but they are serviceable and working and maintainable (if you don't actively block that maintenance). without having any consent with the Debian GCC maintainers, nor announcing these packages anywhere. As you know I have been working on Debian-compatible cross-toolchains since about 2004 on and off. I have discussed them many times with you personally at Connect, Debconf, given talks about them, and of course talked to many other people about them too. There was a thread with a pretty good summary of our basic disagreement here: https://lists.debian.org/debian-devel/2013/01/msg00375.html Most recently we discussed this again at the Bootstrap/Crossbuild Sprint in Paris. In all that time we have never agreed about the best way of doing it, and that disagreement has delayed any upload for the whole of wheezy. No-one uploaded packages using your preferred method in the 4 years they have been used on Ubuntu. I wanted to get some cross-toolchains in Jessie (https://wiki.debian.org/ReleaseGoals/CrossToolchains created Sept 2013) I spent quite some time trying to get the ubuntu cross-toochain-base 'full-bootstrap' packages to build on debian earlier this year but failed. I tried again at Connect in September - still not working. After a couple of weeks I decided it was easier to make the sbuild and wanna-build changes to enable multiarch-built cross-toochains than to fix that mess. So that's what I did (with much help from Dima) so now we do at least have some cross-toolchains. Maintained by a single person, not even trying to form a maintainer group, There is a cross-toolchain maintainer group on Alioth: https://alioth.debian.org/projects/crosstoolchain/ It has existed for a long time, but was re-invigorated at Debconf13 where there were productive cross-toolchain discussions and a clear desire for the above release goal of toolchains in the archive. All the code for the recently uploaded toolchains is there: http://anonscm.debian.org/cgit/crosstoolchain/ (and other packages currently in development, like gcc-cross-support) All those packages are now team-maintained. A few people have been actively working on cross-toolchains over the last 18 months. (Me, Daniel Grosser, Dima Kogan, Helmut Grohne on the linux toolchains, and Keith Packard, Augustin Henze and Thomas Preudhomme on the bare-metal toolchains). There is a team of at least 3 people who have committed to maintaining the existing cross-toolchains (Me, Dima Kogan, Helmut Grohne). I expect that a few more will help now that it has a higher profile in the archive, and indeed YunQiang Su has been helping out recently with mips and multilib changes . You want to replace this team, but who with? and downgrading bug reports without addressing concerns of the GCC maintainers. You filed (#766619, and 5 others for other arches) as 'grave' the fact that the cross-toolchain (providing cpp, gcc, g++ anf gfortran) did not also provide gccgo and gobj. No-one has ever provided crosstoolchains for Debian with those frontends enabled (TTBOMK), so to claim that their abscence is a 'grave' bug seems, well, wrong. So I downgraded it to normal (not even 'wishlist' which is arguably more appropriate). No-one else has asked for these front-ends, and clearly they are not needed for the vast majority of packages. Nevertheless they are quite easy to provide, there is a branch in the cross-gcc git repo that adds those, and I have uploaded them to the cross-toolchain test repo at: http://emdebian.org/tools/debian/. I have not uploaded these to the archive yet because they would have to go through NEW. More front-ends is a fairly minor issue, and the only limitation is how many (each with corresponding meta-package from cross-gcc-defaults) do we want in the archive? Yes. There could have been better communication. However I know that any postings are likely to get an agressive, dismissive, response from you. That is a great way to discourage talking to you.
Bug#771070: requirements for cross toolchain packages in the distribution
On Thu, Dec 18, 2014 at 8:36 AM, Wookey woo...@wookware.org wrote:
MA-built vs in-arch
---
I guess an interesting question is 'what does the cross-compiler
actually _use_ the foreign arch libc for'? Does it need its own
independent copy? What happens when the compiler libc-$arch-cross and
the system libc:$arch get out of sync. Does it matter?
The thought of this gives me nightmares. Glibc is very good at
backwards-compatibility as far as packages go, but does not even
attempt forwards-compatibility.
If anything (I'm still confused if this means only libstdc++, or every
user program - but for those of us writing C++ I suppose it makes no
difference) gets built against libc-$arch-cross (= 2.999) but at
runtime there is only libc:$arch (= 2.998), then programs will almost
certainly fail to load because of missing symbol versions, and
possibly even fail to link.
multilib vs mulitarch
-
Native compilers are not yet co-installable so have to use multilib to
target more than one ABI.
Can we please fix this? I'm tired of having to special-case my
buildbot scripts for arches only available through multilib (i586 and
gnux32 on an amd64 host; this also prevents ever running my buildbot
on anything other than amd64). For all other arches (native or
multiarch cross) the script is trivial.
It does not work to just make my own ~/bin/i586-linux-gnu-gcc etc
scripts that call x86_64-linux-gnu-gcc -m32, because I also have to
worry about the libs being in the wrong place ({/usr,}/lib32 instead
of {/usr,}/lib/i586-linux-gnu, except that it's really
{/usr,}/lib/i386-linux-gnu ... why can't the numbers be consistent?
It's even i486-linux-gnu in places too ... meh, unimportant to this
discussion, except I suppose I'll need to special-case the library
locations anyway, or else lie any call my binaries packages i386 even
though they're really i586 ... it's not like everyone else doesn't
anyway).
Multilibs do make much better sense for arches that are not in the
archive (x32/mipsn32) and are possibly the easier way to support
those, but even there, all the difficulty is in getting the right
libc:$arch or libc-$arch-cross packages. Once you have those you can
build per-arch toolchains or multilib toolchains.
The most correct solution to me seems to be a libs only archive even
for unsupported arches. This would be a huge win even for supported
arches, because 'apt-get update' with N architectures enabled is
really slow already.
Conceptually it would be split into 3 areas that I can think of:
1. anything that ships architecture-specific binaries (not usable in
cross situations in general, but there are still useful packages like
wine32)
2. anything that ships architecture-specific libraries but not
binaries (useful for cross)
3. anything that ships no architecture-specific binaries or libraries
2 is Multi-Arch: same and 3 is Architecture: all, so we already have
an easy way to identify these sets of packages. That said, I can't
think of any case where 1 or 3 would be *used* on their own - 1 deps
on 2 and 3 is only usable with some 1.
Because of packages like wine32, it's impossible to default to only
shipping 2 for non-native arches, so perhaps there could be a second
repo (e.g. http://http.debian.net/debian-libsonly instead of
http://http.debian.net/debian) automatically generated from the main
one (at least for supported arches)? I'm sure there's a way to do this
for x32 packages also (I would be surprised if bootstrapping doesn't
do something like this already, but my familiarity with Debian is as a
user) ... for that matter, in the age of supported cross-gcc, I can
imagine running a future-vanilla debian buildbot for an arch that
doesn't exist yet and using that to populate the initial archive.
There is some upstream expectation of multilib support, which is the
best argument for enabling it. I don't think debian packages should be
using it - they should use gcc-triplet thoughout (then they don't care
how those triplets are packaged-up). We could use a gcc wrapper to
recignise multilib options call the corresponding triplet-gcc. I
beleive this was how guillems 'fake cross-toolchains' worked?
That was my thought too, but I think the important part is backward
compatibility with user build scripts rather than upstream
expectation. If we limited ourselves to upstream expectation,
{/usr,}/$arch/lib wouldn't exist, and we all agree that that's a win,
right? And using that as a guideline, it doesn't have to be *perfect*
backward compatibility, there were certainly packages that needed
patching to deal with it.
-Ben
--
To UNSUBSCRIBE, email to debian-ctte-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive:
https://lists.debian.org/CA+XNFZOrswmPCVLmzYuZWYhEiE=cvv99kyhae+za-aijkby...@mail.gmail.com
Bug#771070: requirements for cross toolchain packages in the distribution
Package: tech-ctte Recently some crippled cross toolchain packages were uploaded to Debian unstable, without having any consent with the Debian GCC maintainers, nor announcing these packages anywhere. Maintained by a single person, not even trying to form a maintainer group, and downgrading bug reports without addressing concerns of the GCC maintainers. I'm asking the ctte to state that - a cross compiler based on GCC within Debian is able to cross build the existing gcc-x.y packages without disabling any features that are offered by the packaging, and used for the native compilers. This includes frontends and multilib'ed runtime libraries. - a cross compiler based on GCC within Debian provides all the front ends required to cross-build GCC. - a cross compiler based on GCC within Debian uses the very same sources, patches and configuration, so that it is equivalent to the native compiler provided by the Debian GCC maintainers. The current maintainer of the cross compiler packages has shown that he is not able to work together on these issues in a group. I'm asking the ctte that anybody who is able and willing to maintain the cross compiler packages as outlined above, becomes the new maintainer of the cross compiler packages targeting Debian architectures. It would be good if the GCC maintainer group is involved in this group. -- To UNSUBSCRIBE, email to debian-ctte-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: https://lists.debian.org/5475e14d.2010...@debian.org
