Re: enabling link time optimizations in package builds
Adrian Bunk: > On Fri, Jun 17, 2022 at 10:18:43AM +0200, Matthias Klose wrote: ... > > >... > > Link time > > optimizations are also at least turned on in other distros like Fedora, > > OpenSuse (two years) and Ubuntu (one year). > >... > > The idea is to file wishlist bug reports for those 373 packages and then see > > how far we get, and if it's feasible to already turn on LTO for bookworm. > > If not, it should be turned on by default for the following release. > > I assume these 373 packages have already been fixed/workarounded in Ubuntu? > Submitting 373 bugs with patches should settle the feasibility question. > For my software (bart), it triggers a compiler bug where the compiler crashes. Martin > A bigger worry is the schedule of such a change. > A major toolchain change shortly before the freeze means the vast > majority of packages will be shipped with non-LTO builds in the release, > with security updates or point release updates triggering a change to > an LTO built package. > This means few packages actually benefitting from LTO, but a higher > regression risk when fixing bugs in stable. > The best timing for such a change would be immediately after the release > of bookworm. >
Re: enabling link time optimizations in package builds
On Fri, Jun 17, 2022 at 12:40:34PM +0300, Nicholas Guriev wrote: > LTO significantly increase memory requirements for buildd machines. Do we > have > enough RAM and swap on each build server? > > > Link time optimizations are also at least turned on in other distros like > > Fedora, OpenSuse (two years) and Ubuntu (one year). > > I know Ubuntu has builders with 8 GB RAM + 4 GB swap which is not enough in > all cases. https://answers.launchpad.net/launchpad/+question/694428 This is on 퐚퐦퐝ퟔퟒ. You can have 24퐓B RAM boxen these days. Assigning 8GB to a buildd is a configuration error -- it's not about building package X at home or porting it to a bitty box. >> The proposal is to turn on LTO by default on most 64bit release >> architectures. As doko CCed -ports: With the hat of one of riscv64 porters on: hardware that is deemed sufficient for buildds has 16 GB RAM and nvme, with very slow CPUs. Thus, please include riscv64 among LTO archs, barring compiler issues of course. Meow! -- ⢀⣴⠾⠻⢶⣦⠀ ⣾⠁⢠⠒⠀⣿⡁ What kind of a drug are "base" and "red pill"? I think acid is ⢿⡄⠘⠷⠚⠋⠀ LSD, which would make base... ? Judging from the behaviour of ⠈⠳⣄ those "based and redpilled", something nasty.
Re: enabling link time optimizations in package builds
On Fri, Jul 01, 2022 at 07:52:16PM +0300, Adrian Bunk wrote: > On Fri, Jun 17, 2022 at 10:18:43AM +0200, Matthias Klose wrote: > > The proposal is to turn on LTO by default on most 64bit release > > architectures. > > By what factor does -ffat-lto-objects increase disk space usage during > package builds? > > Please coordinate with DSA to ensure that the buildds on these > architectures have sufficient diskspace. > > amd64 buildds have/had(?) only 74 GB of diskspace, which has even > without LTO already forced some packages to do manual cleanup steps > during the build to stay within the limited disk space. As amd64 is a fat arch these days, there's no excuse to not simply throw more hardware at this particular problems. We do have excess money and sponsors. > A bigger worry is the schedule of such a change. > A major toolchain change shortly before the freeze means the vast > majority of packages will be shipped with non-LTO builds in the release, So let's do such changes NOW rather than in December. Meow! -- ⢀⣴⠾⠻⢶⣦⠀ ⣾⠁⢠⠒⠀⣿⡁ What kind of a drug are "base" and "red pill"? I think acid is ⢿⡄⠘⠷⠚⠋⠀ LSD, which would make base... ? Judging from the behaviour of ⠈⠳⣄ those "based and redpilled", something nasty.
Re: enabling link time optimizations in package builds
Hi Matthias, * Matthias Klose [2022-06-17 10:18]: The proposal is to turn on LTO by default on most 64bit release architectures. Not proposing to do this on 32bit architectures because of the limited address space at link time, and up to now nobody tested LTO on 32bit archs. In test rebuilds, there were 373 packages (dd-list in the wiki page) found not to build with link time optimizations for various reasons. These range from easily fixable issues in symbols files to some upstream issues. The idea is to fix as many of these as possible, and then change the packaging for the others to just turn off LTO in the package build. In one of my packages, turning on LTO did not fail the build, but exposed a dormant static initialization bug, as LTO apparently messes with the initialization order [1]. I'd expect more bugs of this kind to come up once LTO is enabled by default. Cheers Timo [1] https://github.com/isl-org/Open3D/issues/4747 -- ⢀⣴⠾⠻⢶⣦⠀ ╭╮ ⣾⠁⢠⠒⠀⣿⡁ │ Timo Röhling │ ⢿⡄⠘⠷⠚⠋⠀ │ 9B03 EBB9 8300 DF97 C2B1 23BF CC8C 6BDD 1403 F4CA │ ⠈⠳⣄ ╰╯ signature.asc Description: PGP signature
Re: enabling link time optimizations in package builds
On Fri, Jun 17, 2022 at 10:18:43AM +0200, Matthias Klose wrote: >... > The proposal is to turn on LTO by default on most 64bit release > architectures. >... By what factor does -ffat-lto-objects increase disk space usage during package builds? Please coordinate with DSA to ensure that the buildds on these architectures have sufficient diskspace. amd64 buildds have/had(?) only 74 GB of diskspace, which has even without LTO already forced some packages to do manual cleanup steps during the build to stay within the limited disk space. >... > Link time > optimizations are also at least turned on in other distros like Fedora, > OpenSuse (two years) and Ubuntu (one year). >... > The idea is to file wishlist bug reports for those 373 packages and then see > how far we get, and if it's feasible to already turn on LTO for bookworm. > If not, it should be turned on by default for the following release. I assume these 373 packages have already been fixed/workarounded in Ubuntu? Submitting 373 bugs with patches should settle the feasibility question. A bigger worry is the schedule of such a change. A major toolchain change shortly before the freeze means the vast majority of packages will be shipped with non-LTO builds in the release, with security updates or point release updates triggering a change to an LTO built package. This means few packages actually benefitting from LTO, but a higher regression risk when fixing bugs in stable. The best timing for such a change would be immediately after the release of bookworm. > Matthias cu Adrian
Re: enabling link time optimizations in package builds
LTO significantly increase memory requirements for buildd machines. Do we have enough RAM and swap on each build server? > Link time optimizations are also at least turned on in other distros like > Fedora, OpenSuse (two years) and Ubuntu (one year). I know Ubuntu has builders with 8 GB RAM + 4 GB swap which is not enough in all cases. https://answers.launchpad.net/launchpad/+question/694428 signature.asc Description: This is a digitally signed message part.
enabling link time optimizations in package builds
Link time optimizations are an optimization that helps with a single digit percent number optimizing both for smaller size, and better speed. These optimizations are available for some time now in GCC. Link time optimizations are also at least turned on in other distros like Fedora, OpenSuse (two years) and Ubuntu (one year). Details at https://wiki.debian.org/ToolChain/LTO The proposal is to turn on LTO by default on most 64bit release architectures. Not proposing to do this on 32bit architectures because of the limited address space at link time, and up to now nobody tested LTO on 32bit archs. In test rebuilds, there were 373 packages (dd-list in the wiki page) found not to build with link time optimizations for various reasons. These range from easily fixable issues in symbols files to some upstream issues. The idea is to fix as many of these as possible, and then change the packaging for the others to just turn off LTO in the package build. To explicitly turn on LTO for a package build: export DEB_BUILD_MAINT_OPTIONS=optimize=+lto to explicitly disable LTO: export DEB_BUILD_MAINT_OPTIONS=optimize=-lto The idea is to file wishlist bug reports for those 373 packages and then see how far we get, and if it's feasible to already turn on LTO for bookworm. If not, it should be turned on by default for the following release. Matthias
