On Thu, 2021-08-05 at 17:54 -0400, Tony Battersby wrote: > On 8/5/21 6:21 AM, Richard Purdie wrote: > > On Wed, 2021-08-04 at 20:13 -0400, Randy MacLeod wrote: > > > On 2021-07-27 9:34 a.m., Tony Battersby wrote: > > > > Currently, enabling Link Time Optimization via: > > > > > > > > require conf/distro/include/lto.inc > > > > DISTRO_FEATURES_append = " lto" > > > > > > > > makes the build non-reproducible (as in > > > > https://reproducible-builds.org/). This patch set fixes some of the > > > > problems, but is not yet a complete solution. > > > > > > > > This patch set addresses the following bugs: > > > > https://bugzilla.yoctoproject.org/show_bug.cgi?id=14481 > > > > https://bugzilla.yoctoproject.org/show_bug.cgi?id=14490 > > > > > > > > openembedded-core patches > > > > 1/4: lto.inc: disable LTO for grub > > > > 2/4: gcc: Backport patch to make LTO builds more reproducible > > > > 3/4: bitbake.conf: add DEBUG_PREFIX_MAP to TARGET_LDFLAGS > > > > > > > > openembedded-devel patches > > > > 4/4: net-snmp: fix QA Issue after LDFLAGS change > > > > > > > > Patch 3 adds DEBUG_PREFIX_MAP to TARGET_LDFLAGS (it was already in > > > > TARGET_CFLAGS). This could possibly be done differently, such as by > > > > removing it from TARGET_CFLAGS and adding it to TOOLCHAIN_OPTIONS, > > > > TARGET_CC_ARCH, or TUNE_CCARGS, but I haven't tested those. Let me know > > > > if there is a better/preferred way to do it. > > > > > > > > The TARGET_LDFLAGS change broke net-snmp from meta-networking, so I have > > > > included a patch to fix that. Other packages might have been broken as > > > > well, although I haven't found any others. > > > > > > > > With all these patches, there are still many shared libraries in /lib > > > > and /usr/lib that are still not reproducible with LTO enabled because > > > > x86_64-poky-linux-libtool filters out the -f*-prefix-map flags added by > > > > patch 3 when linking shared libraries. /usr/lib/libbz2.so.1.0.6 from > > > > the libbz2 package from the bzip2 recipe is a good example. I do not > > > > have a patch for that problem. I will leave that to someone else. > > > > > > > > So this is what we need to achieve reproducible LTO builds: > > > > *) This patch set > > > > *) A fix for libtool > > > > *) Add LTO to reproducible builds testing if appropriate (e.g. > > > > meta/lib/oeqa/selftest/cases/reproducible.py, CI on Autobuilder) > > > > *) Fixes for individual packages, if needed > > > > *) Once everything works, update the documentation again (I previously > > > > requested a documentation change for this bug, which will eventually > > > > show up at > > > > http://docs.yoctoproject.org/test-manual/reproducible-builds.html). > > > This is certainly a few steps in the right direction with a good outline > > > of what's next. > > > > > > I suspect that Richard has been busy with other things like overrides > > > and the update to glibc-2.34. > > > > > > Do you have any suggestions or questions about what we should do > > > with this patch series Richard? > > Thanks for the reminder, I've been meaning to queue/test this which I will > > do. The gcc version has been bumped but I ported the patch forward to it > > and the patches are in master-next for testing. It is great to see this > > support moving forward. > > > > Much as it pains me to admit having any libtool knowledge, I may be able to > > help with that bit. There is an untested patch below which may address that > > issue, or at least give a good hint on how to do it if it doesn't work. I > > don't have an environment to test easily right now. > > > > Cheers, > > > > Richard > > I tested your libtool patch in master-next with the bzip2 recipe. It > works as intended; libbz2.so.1.0.6 and all the other output files are > now reproducible with LTO enabled. I also verified that libtool is > passing the -f*-prefix-map flags to the link command in log.do_compile > as intended. > > I also verified that core-image-minimal builds successfully with the > libtool patch.
Thanks for testing that, sounds like it helps that issue. I'll queue that patch. Unfortunately even with the ruby fix I queued, we still see reproduciblity issues with the LDFLAGS change in curl-dev, gettext-ptest, ruby and ruby-dbg packages: https://autobuilder.yoctoproject.org/typhoon/#/builders/115/builds/515/steps/12/logs/stdio We'll need to resolve those before I can merge the flags change. Diffoscope output: https://autobuilder.yocto.io/pub/repro-fail/oe-reproducible-20210805-8jfd55ki/packages/diff-html/ Basically LDFLAGS get encoded into a couple of scripts and the ruby binary. Cheers, Richard
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#154528): https://lists.openembedded.org/g/openembedded-core/message/154528 Mute This Topic: https://lists.openembedded.org/mt/84481652/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
