Hi, In this mail, I'll try to summarize my state of knowledge on this matter while noting that I don't see the full picture.
On Fri, Mar 29, 2024 at 11:17:55AM +0100, Bastian Blank wrote: > On Thu, Mar 21, 2024 at 08:48:01PM +0100, Helmut Grohne wrote: > > I was recently working on gcc builds and this disagreement currently > > makes stuff unbuildable. Hence I looked into solutions and/or > > workarounds. > > Care to just share what you actually found? Where is it broken and how > to see this? > > Because this whole thing started with "it is broken, but I won't tell > you where or what or how". Quite clearly, this is not a single problem, but a mesh of problems and in a few cases it is not obvious where to solve them. > > As a result, I implemented the proposed change and am attaching it for > > discussion here. I've implemented it in a way that if there is a sysroot > > linux header installation, it'll be preferred. Do you see any downsides > > of this approach? > > I wonder now. How would that ever work for the native build? Or does > the native build already do those symlinks? Or are native and cross > configured differently? Or is that a weird difference in gcc itself? Native and cross builds work quite differently indeed. So let me first try to collect all relevant problems that I encountered here. I vaguely try to list the more important ones earlier. I have caused some of these problems and don't want to assign any blame but look for solutions. 1. API expectation of *-$arch-cross packages When *-$arch-cross packages were first introduced before multiarch was a thing, there was (and still is) and implicit contract saying that their functionality is to be provided within the /usr/$DEB_HOST_GNU_TYPE hierarchy. In particular, this layout does not interfere with multiarch on a filesystem level and hence *-$arch-cross packages typically are arch:all m-a:foreign. In particular, linux-libc-dev now provides such packages without actually providing those paths violating this implicit contract. 2. Violation of Multi-Arch: foreign linux-libc-dev was changed from an Arch:any + Multi-Arch:same package to an Architecture:all + Multi-Arch:foreign package. It does so by providing the headers for all architectures in a single package via symlink farms. "all" architectures is debatable though. The set of architectures changes rather frequently with new ones being added and old ones being removed. Therefore, linux-libc-dev will often look like it provides linux-libc-dev:$somearch to dependency resolvers while in fact not providing the functionality - thus violating the promise of Multi-Arch: foreign. For example, linux-libc-dev is currently dysfunctional for arc, but next year it'll be a different architecture. Moreover, looking at the metadata of linux-libc-dev initially did not provide means of figuring out which architectures were actually supported and which were not. This has been changed as linux-libc-dev now Provides linux-libc-dev-$arch-cross packages (causing the first problem), but at least giving bootstrappers a means to figure out whether a given linux-libc-dev package is usable to them. 3. linux-libc-dev consumes much space on mirrors and installations linux-libc-dev originally was Arch:any and yet much of its content was the same across architectures albeit in different paths. Thus, the size of the .deb (multiplied by the number of architectures) was quite big and also coinstalling linux-libc-dev would result in duplicate files being extracted to multiple locations increasing the installation size in a multiarch setting. As a result, linux-libc-dev now is Arch:all and we only get to have one package. It grew from about 1.8MB (times 10 architectures) to about 2.2MB and its installed size grew from about 7MB (per architecture) to 10MB (for all architectures). This change caused the second problem. 4. cross-toolchain-base being bd-uninstallable cross-toolchain-base cannot currently be built (FTBFS #1064003 and #1067370) and one of the aspects is that it declares Build-Conflicts with linux-libc-dev-$arch-cross. The recently added Provides on linux-libc-dev satisfy them and thus cause cross-toolchain-base to be bd-uninstallable. I don't exactly understand why it declares them, but I guess that having a different set of kernel headers available in /usr/$DEB_HOST_GNU_TYPE would cause them to be picked up by the build and potentially cause misbuilds. cross-toolchain-base builds these packages and it also uses them during build of glibc. 5. gcc-V-cross not being buildable The gcc-V-cross package relies on -$arch-cross packages usually built from cross-toolchain-base and expects them to provide their functionality in /usr/$DEB_HOST_GNU_TYPE. The current linux-libc-dev provides the package names but not the expected path (this actually is the first problem) and as a consequence, gcc-V-cross currently fails to build from source. 6. Cross bootstrap cannot tell whether linux-libc-dev supports an architecture Before linux-libc-dev gained the -$arch-cross Provides, there was no way of determining whether linux-libc-dev supports an architecture by looking at its metadata. As a result, cross bootstrap had now way of knowing whether linux-libc-dev needed to be rebuilt. This is a consequence of the second problem. 7. Cross bootstrap needs to deal with Arch:all packages Until linux-libc-dev became an architecture-dependent Arch:all package, the entire cross bootstrap was possible by only performing arch-only builds and using a repository of only arch:any packages used in conjunction with a Debian mirror restricted to Arch:all packages. Now, the bootstrap repository must also be able to carry an Arch:all package and handle the fact that multiple versions of linux-libc-dev exist in a bootstrap setting one of which does not work (as a result of the second problem). 8. Duplication of functionality via -$arch-cross packages When performing cross builds, we currently need both -$arch-cross packages and :$arch packages for glibc and linux-libc-dev. These can be built from different versions and we know that using libc6-dev-$arch-cross packages built from a different glibc version than libc6-dev:$arch together causes problems (repeatedly) and hence glibc now declares Conflicts with old libc6-dev-$arch-cross packages. If gcc-V-cross were to use :$arch packages, it would have to declare cross-architecture dependencies, which is not currently supported by our buildd network. That is one reason for it still using -$arch-cross packages. When I say problems here, you may read that more as "challenge" and in particular, it is not obvious that the change that caused a problem should be reverted. Rather, there are often multiple ways to resolve these. Let me sketch a few options that may not be obvious from this list: 1+3+5. linux-libc-dev could add a symlink farm for /usr/$DEB_HOST_GNU_TYPE. 1+3+4+6. linux-libc-dev could rename its Provides dropping the -cross suffix. 2. Ignore the violation. libc6 also violates Multi-Arch:same, because the dynamic loader conflicts for some pairs of architectures. There is nothing that can be done about this as architecture-dependent breaks or conflicts are not supported. Hence, we can argue that this violation is so useful that it should be tolerated. 2+3+6+7. linux-libc-dev could be split into linux-libc-dev-common arch:all m-a:foreign and the symlink farms could be kept in linux-libc-dev:any m-a:same retaining the size reduction. 3. cross-toolchain-base could build a linux-libc-dev-cross package that Provides the earlier -$arch-cross packages and contains a similar symlink farm to linux-libc-dev. 3+8. cross-toolchain-base could stop creating linux-libc-dev-$arch-cross packages. Patch: https://bugs.debian.org/1059786#10 4. cross-toolchain-base could drop its Build-Conflicts. This may cause further problems though. Patch: https://bugs.debian.org/1067370#17 5. gcc-V-cross could create its own symlink farm for linux-libc-dev at build time and pass that to the gcc build removing the need for the /usr/$DEB_HOST_GNU_TYPE location. Patch: https://bugs.debian.org/1065416#129 7. cross bootstrap can implement indep builds. 8. Implement cross-architecture dependencies in the buildd network such that we can get rid of -$arch-cross packages. Please consider both the list of problems and in particular the list of options incomplete. On a procedural level, linux-libc-dev hijacked linux-libc-dev-$arch-cross. After I sent https://bugs.debian.org/1059786#10, there was quite some silence which I mistakenly interpreted as agreement and hence suggested to Bastian that linux-libc-dev should provide linux-libc-dev-$arch-cross as a solution to the sixth problem. Regarding CTTE, I am prejudiced on this matter and want to abstain from voting. Helmut
