Steffen Möller writes ("And in 2019? Re: -flto to become more of a routine -
any change in opinion since 2011?"):
> We just had SuSE embracing LTO
> (https://www.linuxtoday.com/infrastructure/opensuse-enables-lto-by-default-for-tumbleweed-smaller-faster-binaries.html).
> I am not sure about the progress on issues summarised in
> http://blog.regehr.org/archives/1180 that Ian pointed to. But since I
> last asked in 2016 we have more pedantic compiler settings and more CI -
> and LTO, as much as compilers have improved on that, does not need to be
> applied everywhere. Any change in opinion?
Thanks for asking. I think this needs to be dealt with by way of risk
assessment.
Within a particular source package or maybe within a particular
ecosystem, it may be possible to make an argument that there are no
(or few) new misoptimisation opportunities.
Across the distribution as a whole I think it is far too risky unless
we get some buy-in from compiler folks and a way to limit the kind of
things the compiler might do.
The biggest problem with LTO is this:
We design our whole system, and particularly C code, with knowledge of
how compilers translate source code for externally-visible objects
into a concrete ABI. (This is even documented, of course, in ABI
specifications.)
With knowledge of that ABI, we often make changes where we link
together code which has a compatible ABI but where the source code
expression of the interface is not identical.
But C's rules for what functions and structs and so on are equivalent
enough to call across translation unit boundaries are much much
stricter than the rules implied by the ABI specifications.
Now so far this seems OK because dynamic linking means that we aren't
actually doing LTO across these ABI boundaries (yet). However, of
course, we sometimes link things statically.
Without LTO the header files can describe the ABI and don't need to
exactly match, according to the stricter C rules, the definitions of
the functions. With LTO the header files must match exactly.
Making the header files match exactly sounds easy because it seems
we're talking now about things in the same source package. But:
* Sometimes different source packages intend to provide
the same ABI and then the headers might be in a different
source package.
* Authors of libraries often want to write forward- or backward-
compatibility arrangements in the headers. When you do this
it is often convenient for build system reasons to build
some parts of the system with some set of compatibility #defines
or whatever and some parts with a different set. This is
OK so long as the ABI is right - until LTO.
* Other scenarios I haven't thought of right now.
Basically, before LTO we could all assume that cross-translation-unit
compatibility was determined entirely by ABI rules. These ABI rules
are comprehensible, relatively flexible, roughly equivalent in
implications across different architectures (at least in the commonly
used subset), and embedded both in our body of existing software and
in our programming habits.
With LTO, the ABI rules go out of the window whenever LTO applies and
instead we must satisfy the C standard rules. The C standard rules
are obtuse, insanely complicated, and generally hard to satisfy or
even analyse in nontrivial cases. Textually-(nearly)-identical
declarations suffice of course, but it is often necessary or desirable
to effectively use the declaration to specify the ABI and then rely on
ABI compatibility.
I hope this explanation is helpful.
Ian.
