Re: Fedora 32 System-Wide Change proposal: LTO by default for package builds
On Fri, 2019-12-20 at 11:52 +0100, Miro Hrončok wrote: > On 19. 12. 19 22:41, Ben Cotton wrote: > > https://fedoraproject.org/wiki/LTOByDefault > > > > == Contingency Plan == > > * Contingency mechanism: Revert the LTO flags injection > > * Contingency deadline: Beta freeze, but shooting for prior to mass > > rebuilds starting > > * Blocks release? No > > * Blocks product? No > > > > Most critically, if we don't address the GDB testsuite issue noted > > above, our fallback position would be to simply disable the LTO > > injection globally and re-evaluate for Fedora 33, similarly if we were > > to find some show-stopping LTO issue. > > Should the contingency plan include a second mass rebuild in case our > packages > successfully built with lto during the mass rebuild, but are broken at > runtime? > > Or do we safely assume that it's good as long as it builds fine? It would really depend on what we find and it's the kind of decision we deal with semi-regularly with the yearly compiler update in Fedora -- what to do if we find something horrifically wrong, whether it be an unexpected ABI change or a particularly nasty codegen issue. When we encounter these situations we often end up instrumenting the compiler to get a sense of how pervasive the problem is or building a scanner to look for problematic code sequences across the builds done with the problematic compiler/options. To-date I don't think we've ever had to do something like back out a major compiler update or request a mass rebuild in Fedora as a result of a compiler issue. We've had some "phew" moments for issues we thought were going to have wide impact, but for various reasons didn't. So while I wish I could give you a concrete answer, it's really something we (Jakub, Marek, Jason, Jon, myself and others) evaluate on a case by case basis looking at size/scope and determine a path forward, knowing that an unexpected mass rebuild would be catastrophic. What gives me a relatively high degree of confidence in the introduction of LTO is that openSUSE made this change for their distro in the spring of this year confirming the base LTO technology on a wide scale, combined with the testing we're already doing for Fedora with gcc-10. By *far* the biggest issue with LTO is the configure test snippets that are so horribly written that they can be easily compromised by aggressive optimization enabled by LTO and the widespread nature of that horrific code. Hence the desire to fix the most common and egregious issues by in-place editing of the already-generated configure files within %configure. And just to be clear, these code snippets would never actually run and nobody would ever write code this way expecting it to be executed -- they're primarily compile/link tests to look for features. Concerns over mis-configuring packages due to these issues is what led to the introduction of the capability to capture the generated config.h files and compare them against builds with standard tools in our tester. If we note any differences in the generated config.h files, the build is considered a failure and needs to be examined and the issue fixed. jeff ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Fedora 32 System-Wide Change proposal: LTO by default for package builds
On 19. 12. 19 22:41, Ben Cotton wrote: https://fedoraproject.org/wiki/LTOByDefault == Contingency Plan == * Contingency mechanism: Revert the LTO flags injection * Contingency deadline: Beta freeze, but shooting for prior to mass rebuilds starting * Blocks release? No * Blocks product? No Most critically, if we don't address the GDB testsuite issue noted above, our fallback position would be to simply disable the LTO injection globally and re-evaluate for Fedora 33, similarly if we were to find some show-stopping LTO issue. Should the contingency plan include a second mass rebuild in case our packages successfully built with lto during the mass rebuild, but are broken at runtime? Or do we safely assume that it's good as long as it builds fine? -- Miro Hrončok -- Phone: +420777974800 IRC: mhroncok ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Fedora 32 System-Wide Change proposal: LTO by default for package builds
On 19.12.2019 22:41, Ben Cotton wrote: > This is a proposal to enable link time optimization (LTO) of packages > built with rpmbuild by default. It's an over-simplification, but > think of LTO as deferring analysis, optimization and code generation > until creation of an executable or dynamic shared object. Nice. LTO can significantly reduce binary size. I tested it on my packages and it work great except Qt5 packages built with clang compiler due to upstream bug[1]. Currently even Qt's hello world app cannot be compiled by clang with LTO enabled. [1]: https://bugreports.qt.io/browse/QTBUG-61710 -- Sincerely, Vitaly Zaitsev (vit...@easycoding.org) ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Fedora 32 System-Wide Change proposal: LTO by default for package builds
Hi Jeff, On Thu, Dec 19, 2019 at 10:50 PM Ben Cotton wrote: > > https://fedoraproject.org/wiki/LTOByDefault > > == Summary == > This is a proposal to enable link time optimization (LTO) of packages > built with rpmbuild by default. It's an over-simplification, but > think of LTO as deferring analysis, optimization and code generation > until creation of an executable or dynamic shared object. > > This is implemented by adding the option "-flto" the injected flags in > redhat-rpm-config. There will be a simple way for packages to opt-out > of LTO. > > == Owner == > * Name: Jeff Law > * Email: l...@redhat.com > > == Detailed Description == > Programs built with rpmbuild and which honor flags injection via > redhat-rpm-config will be built with LTO by default. A simple opt-out > mechanism will be provided for packages which use features that are > not LTO compatible. > > The LTO bytecode itself will not be distributed as it is not stable > from one GCC release to the next. This is enforced by stripping the > LTO bytecode from any installed .o/.a files. We'll use bits SuSE has > already written for redhat-rpm-config to implement this. > > Minor changes are desirable to the %configure macro in > redhat-rpm-config to fix common code idioms used by autoconf generated > scripts which are compromised by the additional optimization enabled > by LTO. Minor updates to various packages will be needed to opt-out > of LTO or fix bugs exposed by LTO. > > == Benefit to Fedora == > > The primary benefits of building with LTO enabled are smaller, faster > executables/DSOs. A secondary benefit is LTO allows deeper analysis > of package source code at compile time which can improve various GCC > diagnostics and thus improve our ability to catch bugs at compile time > such as uninitialized objects, buffer overflows, unterminated strings, > restrict violations, etc. It would be very nice to get more specific analysis data. Like running some benchmarks of big applications, size comparisons (of binaries and libraries) and compile time. > This change also brings us back on-par with SuSE who enabled LTO by > default for their free distribution earlier in 2019. You probably should have said openSUSE rather here. > > == Scope == > * Proposal owners: > The primary change is to redhat-rpm-config to add LTO to the default > compile/link flags as well as a conditional which allows easy opt-out > on a package by package basis. Additionally the post-build scripts > need to strip the LTO bytecodes from any installed .o/.a files. What does that mean? Which post-build scripts are needed and where that needs to be done? How does it affect build time? > Additionally, we know there are many packages with configure scripts > that are compromised by LTO. I have tweaks to the %configure macro in > redhat-rpm-config which fixes the vast majority of these problems with > a few simple sed scripts on the generated output. Like the basic > support for injecting the LTO flags, this will require coordination > with the redhat-rpm-config maintainers. Packages which call configure > directly and have compromised tests will need a one line change to > their .spec files to fix their configure scripts. "simple sed scripts" are probably not so simple :) > Some packages will need to opt-out of using LTO at this time. The > most common case are packages that use symbol versioning or toplevel > ASM statements. While there is a new mechanism to make LTO work with > symbol versioning, I don't think any packages have been updated to use > that mechanism. This will require a one line change to 50-75 packages > (my script to find these is still running). Hmm, I think we have bunch of packages (more than 75 of them) which use symbol versioning. Would be useful to give some links in the change proposal to "how to update to use that mechanism". > Finally, some packages will fail to build with LTO due to deeper > analysis for compile-time diagnostics catching programming mistakes > that have gone unnoticed until now. I'll obviously be working with > package maintainers on all of these issues. > > Note that even though the changes are fairly well localized in > redhat-rpm-config and a small number of packages, the real scope of > this change is much larger since it affects all packages in the > distribution that are compiled with GCC and which honor the flags > injection by redhat-rpm-config. > > > * Other developers: > As I mentioned, I'm happy to contact package owners that need to > modify their packages and suggest how their package needs to be fixed. > As a multi-decade GCC developer, I'm particularly well suited to > describe LTO, its limitations and how LTO impacts the diagnostics from > GCC to any package owner that needs additional information. > > I'm also capable and available to address any GCC issues that we may > arise as a result of this change. I don't expect much of the latter > as SuSE has already enabled this feature for their d
Re: Fedora 32 System-Wide Change proposal: LTO by default for package builds
On Thu, Dec 19, 2019 at 3:42 PM Ben Cotton wrote: > > https://fedoraproject.org/wiki/LTOByDefault > > == Summary == > This is a proposal to enable link time optimization (LTO) of packages > built with rpmbuild by default. It's an over-simplification, but > think of LTO as deferring analysis, optimization and code generation > until creation of an executable or dynamic shared object. > > This is implemented by adding the option "-flto" the injected flags in > redhat-rpm-config. There will be a simple way for packages to opt-out > of LTO. > > == Owner == > * Name: Jeff Law > * Email: l...@redhat.com > > == Detailed Description == > Programs built with rpmbuild and which honor flags injection via > redhat-rpm-config will be built with LTO by default. A simple opt-out > mechanism will be provided for packages which use features that are > not LTO compatible. > > The LTO bytecode itself will not be distributed as it is not stable > from one GCC release to the next. This is enforced by stripping the > LTO bytecode from any installed .o/.a files. We'll use bits SuSE has > already written for redhat-rpm-config to implement this. > > Minor changes are desirable to the %configure macro in > redhat-rpm-config to fix common code idioms used by autoconf generated > scripts which are compromised by the additional optimization enabled > by LTO. Minor updates to various packages will be needed to opt-out > of LTO or fix bugs exposed by LTO. > > == Benefit to Fedora == > > The primary benefits of building with LTO enabled are smaller, faster > executables/DSOs. A secondary benefit is LTO allows deeper analysis > of package source code at compile time which can improve various GCC > diagnostics and thus improve our ability to catch bugs at compile time > such as uninitialized objects, buffer overflows, unterminated strings, > restrict violations, etc. > > This change also brings us back on-par with SuSE who enabled LTO by > default for their free distribution earlier in 2019. > > > == Scope == > * Proposal owners: > The primary change is to redhat-rpm-config to add LTO to the default > compile/link flags as well as a conditional which allows easy opt-out > on a package by package basis. Additionally the post-build scripts > need to strip the LTO bytecodes from any installed .o/.a files. > > Additionally, we know there are many packages with configure scripts > that are compromised by LTO. I have tweaks to the %configure macro in > redhat-rpm-config which fixes the vast majority of these problems with > a few simple sed scripts on the generated output. Like the basic > support for injecting the LTO flags, this will require coordination > with the redhat-rpm-config maintainers. Packages which call configure > directly and have compromised tests will need a one line change to > their .spec files to fix their configure scripts. > > Some packages will need to opt-out of using LTO at this time. The > most common case are packages that use symbol versioning or toplevel > ASM statements. While there is a new mechanism to make LTO work with > symbol versioning, I don't think any packages have been updated to use > that mechanism. This will require a one line change to 50-75 packages > (my script to find these is still running). > > Finally, some packages will fail to build with LTO due to deeper > analysis for compile-time diagnostics catching programming mistakes > that have gone unnoticed until now. I'll obviously be working with > package maintainers on all of these issues. > > Note that even though the changes are fairly well localized in > redhat-rpm-config and a small number of packages, the real scope of > this change is much larger since it affects all packages in the > distribution that are compiled with GCC and which honor the flags > injection by redhat-rpm-config. > > > * Other developers: > As I mentioned, I'm happy to contact package owners that need to > modify their packages and suggest how their package needs to be fixed. > As a multi-decade GCC developer, I'm particularly well suited to > describe LTO, its limitations and how LTO impacts the diagnostics from > GCC to any package owner that needs additional information. > > I'm also capable and available to address any GCC issues that we may > arise as a result of this change. I don't expect much of the latter > as SuSE has already enabled this feature for their distribution and > thus weeded out most of the issues. > > The highest level of coordination will be with the redhat-rpm-config > maintainers. > > I will also be coordinating with the GDB team to address debugging > issues related to LTO. The most important issue is to ensure that we > can pass the GDB testsuite with and without the -flto option being > enabled.Failure to meet this goal would be considered a blocking > issue for LTO enablement. > > I'm also already in contact with SuSE and Debian/Ununtu engineers to > discuss issues with gcc-10 with and without LTO. > > We know there ar
Fedora 32 System-Wide Change proposal: LTO by default for package builds
https://fedoraproject.org/wiki/LTOByDefault == Summary == This is a proposal to enable link time optimization (LTO) of packages built with rpmbuild by default. It's an over-simplification, but think of LTO as deferring analysis, optimization and code generation until creation of an executable or dynamic shared object. This is implemented by adding the option "-flto" the injected flags in redhat-rpm-config. There will be a simple way for packages to opt-out of LTO. == Owner == * Name: Jeff Law * Email: l...@redhat.com == Detailed Description == Programs built with rpmbuild and which honor flags injection via redhat-rpm-config will be built with LTO by default. A simple opt-out mechanism will be provided for packages which use features that are not LTO compatible. The LTO bytecode itself will not be distributed as it is not stable from one GCC release to the next. This is enforced by stripping the LTO bytecode from any installed .o/.a files. We'll use bits SuSE has already written for redhat-rpm-config to implement this. Minor changes are desirable to the %configure macro in redhat-rpm-config to fix common code idioms used by autoconf generated scripts which are compromised by the additional optimization enabled by LTO. Minor updates to various packages will be needed to opt-out of LTO or fix bugs exposed by LTO. == Benefit to Fedora == The primary benefits of building with LTO enabled are smaller, faster executables/DSOs. A secondary benefit is LTO allows deeper analysis of package source code at compile time which can improve various GCC diagnostics and thus improve our ability to catch bugs at compile time such as uninitialized objects, buffer overflows, unterminated strings, restrict violations, etc. This change also brings us back on-par with SuSE who enabled LTO by default for their free distribution earlier in 2019. == Scope == * Proposal owners: The primary change is to redhat-rpm-config to add LTO to the default compile/link flags as well as a conditional which allows easy opt-out on a package by package basis. Additionally the post-build scripts need to strip the LTO bytecodes from any installed .o/.a files. Additionally, we know there are many packages with configure scripts that are compromised by LTO. I have tweaks to the %configure macro in redhat-rpm-config which fixes the vast majority of these problems with a few simple sed scripts on the generated output. Like the basic support for injecting the LTO flags, this will require coordination with the redhat-rpm-config maintainers. Packages which call configure directly and have compromised tests will need a one line change to their .spec files to fix their configure scripts. Some packages will need to opt-out of using LTO at this time. The most common case are packages that use symbol versioning or toplevel ASM statements. While there is a new mechanism to make LTO work with symbol versioning, I don't think any packages have been updated to use that mechanism. This will require a one line change to 50-75 packages (my script to find these is still running). Finally, some packages will fail to build with LTO due to deeper analysis for compile-time diagnostics catching programming mistakes that have gone unnoticed until now. I'll obviously be working with package maintainers on all of these issues. Note that even though the changes are fairly well localized in redhat-rpm-config and a small number of packages, the real scope of this change is much larger since it affects all packages in the distribution that are compiled with GCC and which honor the flags injection by redhat-rpm-config. * Other developers: As I mentioned, I'm happy to contact package owners that need to modify their packages and suggest how their package needs to be fixed. As a multi-decade GCC developer, I'm particularly well suited to describe LTO, its limitations and how LTO impacts the diagnostics from GCC to any package owner that needs additional information. I'm also capable and available to address any GCC issues that we may arise as a result of this change. I don't expect much of the latter as SuSE has already enabled this feature for their distribution and thus weeded out most of the issues. The highest level of coordination will be with the redhat-rpm-config maintainers. I will also be coordinating with the GDB team to address debugging issues related to LTO. The most important issue is to ensure that we can pass the GDB testsuite with and without the -flto option being enabled.Failure to meet this goal would be considered a blocking issue for LTO enablement. I'm also already in contact with SuSE and Debian/Ununtu engineers to discuss issues with gcc-10 with and without LTO. We know there are some problems with debugging LTO code. I will be working with the GDB team to identify these issues and fix them either in the debugger or compiler as needed. I have prototype code for the required redhat-rpm-config changes and I'll coordi
