Bug#930997: cmake: reprotest's 'setarch linux32 dh_auto_configure' on amd64 results in CMAKE_SYSTEM_PROCESSOR = i386
Control: reassign -1 reprotest Control: tags -1 -moreinfo Helmut Grohne: > Hi Niels, > > On Sat, Sep 14, 2019 at 06:08:00AM +, Niels Thykier wrote: >> Is there any update on this? At the moment, this bug is not actionable >> and sprayed over three packages. > > No. I don't think there are any news here. I had hoped that my previous > mails would have made my position clear. Let me try to be more explicit: > I think that running native amd64 builds in linux32 is broken and that > any tool doing so (e.g. reprotest) is buggy. > > Having debhelper pass CMAKE_SYSTEM_PROCESSOR is a nice idea, but doing > so will break a fair number of packages in subtle ways. Therefore I > recommend not doing that (other than for cross compilation). > > This is basically repeating what I already said. If you concur, the > logical next step is reassigning to reprotest. > > Helmut > Thanks for clarifying; I missed that part. Reassigning to reprotest Thanks, ~Niels
Bug#930997: cmake: reprotest's 'setarch linux32 dh_auto_configure' on amd64 results in CMAKE_SYSTEM_PROCESSOR = i386
Hi Niels, On Sat, Sep 14, 2019 at 06:08:00AM +, Niels Thykier wrote: > Is there any update on this? At the moment, this bug is not actionable > and sprayed over three packages. No. I don't think there are any news here. I had hoped that my previous mails would have made my position clear. Let me try to be more explicit: I think that running native amd64 builds in linux32 is broken and that any tool doing so (e.g. reprotest) is buggy. Having debhelper pass CMAKE_SYSTEM_PROCESSOR is a nice idea, but doing so will break a fair number of packages in subtle ways. Therefore I recommend not doing that (other than for cross compilation). This is basically repeating what I already said. If you concur, the logical next step is reassigning to reprotest. Helmut
Bug#930997: cmake: reprotest's 'setarch linux32 dh_auto_configure' on amd64 results in CMAKE_SYSTEM_PROCESSOR = i386
On Wed, 14 Aug 2019 17:24:00 + Niels Thykier wrote: > Control: tags -1 moreinfo > Hi Simon and Helmut, Is there any update on this? At the moment, this bug is not actionable and sprayed over three packages. (last reply quoted in full so you don't have to look up the bug) Thanks, ~Niels > Simon McVittie: > > On Sun, 14 Jul 2019 at 08:43:25 +0200, Helmut Grohne wrote: > >> [...] > >> Other than that, my general advice would be preferring $CC -dumpmachine > >> over uname -m as it avoids a whole host of problems. Getting there seems > >> like a herculean task though. > > > > Is -dumpmachine portable among compilers, or is it a GNU'ism? If it's > > specific to gcc, or specific to gcc and compilers like clang that mimic > > gcc, or specific to compilers designed with the GNU/Autoconf vocabulary > > of CPUs in mind, then I can see why upstreams targeting both GNU and > > non-GNU OSs would avoid it. > > > > As far as I can tell, CMake uses uname -m for its vocabulary of Linux CPUs > > (but see https://bugs.debian.org/930995), so switching to using the CPU > > part of $CC -dumpmachine would be an incompatible change, unless CMake > > had a lookup table to map between GNU CPUs and what uname -m would have > > said on the relevant machine. I think Meson did this better by having an > > explicitly documented table of known CPU names; I would have preferred it > > if Meson had reused GNU's vocabulary of CPU names rather than inventing > > a new one, but it's too late for that. > > > > Debian::Debhelper::Buildsystem::cmake effectively already does have > > a lookup table to map GNU CPUs to uname -m (it's a list of exceptions > > rather than a complete table, since in practice they usually match), > > but it's currently only used when told to cross-compile, and reprotest's > > builds are "officially" not cross-compiling, even if uname -m would > > indicate otherwise. > > > > smcv > > > > Hi, > > I am not sure I am any wiser on this bug after reading the bug log other > than it looks unactionable to me at the moment (hench the moreinfo tag). > > If the conclusion is that debhelper should always pass > -DCMAKE_SYSTEM_PROCESSOR (or/and -DCMAKE_SYSTEM_NAME) then I am happy to > implement that after we confirmed that this is "safe" (doesn't break > every cmake package in sid, etc.). However, it is not clear to me > whether that is the conclusion we reached. > > Thanks, > ~Niels >
Bug#930997: cmake: reprotest's 'setarch linux32 dh_auto_configure' on amd64 results in CMAKE_SYSTEM_PROCESSOR = i386
Control: tags -1 moreinfo Simon McVittie: > On Sun, 14 Jul 2019 at 08:43:25 +0200, Helmut Grohne wrote: >> [...] >> Other than that, my general advice would be preferring $CC -dumpmachine >> over uname -m as it avoids a whole host of problems. Getting there seems >> like a herculean task though. > > Is -dumpmachine portable among compilers, or is it a GNU'ism? If it's > specific to gcc, or specific to gcc and compilers like clang that mimic > gcc, or specific to compilers designed with the GNU/Autoconf vocabulary > of CPUs in mind, then I can see why upstreams targeting both GNU and > non-GNU OSs would avoid it. > > As far as I can tell, CMake uses uname -m for its vocabulary of Linux CPUs > (but see https://bugs.debian.org/930995), so switching to using the CPU > part of $CC -dumpmachine would be an incompatible change, unless CMake > had a lookup table to map between GNU CPUs and what uname -m would have > said on the relevant machine. I think Meson did this better by having an > explicitly documented table of known CPU names; I would have preferred it > if Meson had reused GNU's vocabulary of CPU names rather than inventing > a new one, but it's too late for that. > > Debian::Debhelper::Buildsystem::cmake effectively already does have > a lookup table to map GNU CPUs to uname -m (it's a list of exceptions > rather than a complete table, since in practice they usually match), > but it's currently only used when told to cross-compile, and reprotest's > builds are "officially" not cross-compiling, even if uname -m would > indicate otherwise. > > smcv > Hi, I am not sure I am any wiser on this bug after reading the bug log other than it looks unactionable to me at the moment (hench the moreinfo tag). If the conclusion is that debhelper should always pass -DCMAKE_SYSTEM_PROCESSOR (or/and -DCMAKE_SYSTEM_NAME) then I am happy to implement that after we confirmed that this is "safe" (doesn't break every cmake package in sid, etc.). However, it is not clear to me whether that is the conclusion we reached. Thanks, ~Niels
Bug#930997: cmake: reprotest's 'setarch linux32 dh_auto_configure' on amd64 results in CMAKE_SYSTEM_PROCESSOR = i386
On Sun, 14 Jul 2019 at 08:43:25 +0200, Helmut Grohne wrote: > Performing a native amd64 build with linux32 sounds seriously broken to > me. ... > However, the reverse seems to be somewhat tolerated: Performing a native > i386 build in an i386 chroot on an amd64 kernel is commonly expected to > work. In this setting, a linux32 variation makes somewhat sense. Yes, the policy I would have expected goes something like this: * packages MUST do a successful non-cross build if $(uname -m) agrees with the dpkg host architecture (dpkg says i386 and uname says i[3456]86, etc.) * packages SHOULD do a successful non-cross build if $(uname -m) indicates anything "better than" the baseline for the dpkg architecture (where x86_64 > i386, mips64el > mipsel, arm64 > armhf > armel and so on) * packages are not required to do a successful non-cross build if $(uname -m) indicates something "worse than" or incompatible with the baseline for the dpkg architecture (e.g. dpkg says amd64 and uname says i[3456]86, or dpkg says ppc64 and uname says ppc64el or powerpc) > As far as I know, the official buildds do linux32 for i386 to avoid > problems. Yes. If it was up to me, I would recommend for official/production buildds (which just want working binaries or a RC bug report) to wrap builds for all 32-bit architectures in linux32. I think it's OK for QA builds like reprotest, which want pedantic correctness more than working code, to try doing 32-bit non-cross builds in a 32-bit chroot on a 64-bit build machine without linux32, as long as it's understood that the resulting bug report on failure might not be RC. > Other than that, my general advice would be preferring $CC -dumpmachine > over uname -m as it avoids a whole host of problems. Getting there seems > like a herculean task though. Is -dumpmachine portable among compilers, or is it a GNU'ism? If it's specific to gcc, or specific to gcc and compilers like clang that mimic gcc, or specific to compilers designed with the GNU/Autoconf vocabulary of CPUs in mind, then I can see why upstreams targeting both GNU and non-GNU OSs would avoid it. As far as I can tell, CMake uses uname -m for its vocabulary of Linux CPUs (but see https://bugs.debian.org/930995), so switching to using the CPU part of $CC -dumpmachine would be an incompatible change, unless CMake had a lookup table to map between GNU CPUs and what uname -m would have said on the relevant machine. I think Meson did this better by having an explicitly documented table of known CPU names; I would have preferred it if Meson had reused GNU's vocabulary of CPU names rather than inventing a new one, but it's too late for that. Debian::Debhelper::Buildsystem::cmake effectively already does have a lookup table to map GNU CPUs to uname -m (it's a list of exceptions rather than a complete table, since in practice they usually match), but it's currently only used when told to cross-compile, and reprotest's builds are "officially" not cross-compiling, even if uname -m would indicate otherwise. smcv
Bug#930997: cmake: reprotest's 'setarch linux32 dh_auto_configure' on amd64 results in CMAKE_SYSTEM_PROCESSOR = i386
Hi Simon, On Mon, Jun 24, 2019 at 11:28:42AM +0100, Simon McVittie wrote: > - maybe reprotest shouldn't be using linux32 to build packages for amd64 > when DEB_BUILD_ARCH_CPU and DEB_HOST_ARCH_CPU are both amd64, because > this causes uname(2) and DEB_BUILD_ARCH_CPU to be inconsistent, > and causes debhelper's is_cross_compiling() to return false, which > makes debhelper assume it doesn't need to override things like > CMAKE_SYSTEM_PROCESSOR? Performing a native amd64 build with linux32 sounds seriously broken to me. There are very many (non-cmake) packages that rely on uname -m and many of them will misbuild in such a setting. Of course, every such use of uname -m needs to be worked around for cross building. In a perfect world, build systems wouldn't be using uname -m at all. This is not where we are today. However, the reverse seems to be somewhat tolerated: Performing a native i386 build in an i386 chroot on an amd64 kernel is commonly expected to work. In this setting, a linux32 variation makes somewhat sense. A number of build systems (including meson) have code to handle this. As far as I know, the official buildds do linux32 for i386 to avoid problems. So I'd first like to understand the rationale for this reprotest behaviour. Other than that, my general advice would be preferring $CC -dumpmachine over uname -m as it avoids a whole host of problems. Getting there seems like a herculean task though. Helmut
Bug#930997: cmake: reprotest's 'setarch linux32 dh_auto_configure' on amd64 results in CMAKE_SYSTEM_PROCESSOR = i386
Package: cmake,debhelper,reprotest Version: 3.13.4-1,12.1.1,0.7.8 Severity: normal I tried enabling the Salsa CI team pipelines[1] for openjk[2] and found an unexpected result: in the second of two builds done by reprotest, openjk is compiled with an x86_64 compiler but appears to have CMAKE_SYSTEM_PROCESSOR (the equivalent of DEB_HOST_ARCH_CPU) set to i386. In openjk this results in the loadable modules (which have architecture-specific names) being named differently, causing extensive binary differences. I can't be sure (it isn't currently logged by the salsa-ci-team pipline[3]) but I think reprotest is using 'setarch i386', equivalent to linux32(1), for the second build. I don't know which of the packages involved is at fault here, but this is surely a problem with at least one of them: - maybe reprotest shouldn't be using linux32 to build packages for amd64 when DEB_BUILD_ARCH_CPU and DEB_HOST_ARCH_CPU are both amd64, because this causes uname(2) and DEB_BUILD_ARCH_CPU to be inconsistent, and causes debhelper's is_cross_compiling() to return false, which makes debhelper assume it doesn't need to override things like CMAKE_SYSTEM_PROCESSOR? - or maybe debhelper's cmake build system should be forcing CMake to set [CMAKE_HOST_SYSTEM_PROCESSOR to a value appropriate for the DEB_BUILD_ARCH_CPU and?] CMAKE_SYSTEM_PROCESSOR to a value appropriate for the DEB_HOST_ARCH_CPU even when is_cross_compiling() returns false, so that CMake will obey those settings instead of calling uname(2)? - or maybe it's CMake that should be doing something differently? A similar situation would presumably appear if you used reprotest on other 64-bit architectures that have a non-default 32-bit personality, like aarch64/arm, powerpc64/powerpc and mips64el/mipsel. Complicating factors: * CMake cross-compiling terminology is not the same as is used in dpkg, so you can't usefully say "host" when discussing this issue without clarifying whose definition of host you are using. dpkg uses the GNU terminology (as seen in Autotools and Meson), where you compile on the build architecture, producing binaries suitable to be run on the host architecture (which might themselves in rare cases be cross-compilers that produce code for the target architecture). In CMake terminology, you compile on the host system (which is the GNU build system), producing binaries suitable to be run on the target system (which is the GNU host system). * CMake doesn't document its taxonomy of CPUs (unlike for example dpkg, GNU and Meson, which each have a (subtly different!) canonical list), and it appears that it will be different on different OSs; but at least on Linux (and possibly GNU/kFreeBSD and GNU/Hurd?) it appears that in practice it uses (struct utsname).machine, which is the same thing as Linux uname -m. [1] https://salsa.debian.org/salsa-ci-team/pipeline/ [2] https://tracker.debian.org/pkg/openjk [3] https://salsa.debian.org/salsa-ci-team/pipeline/issues/56
