On 8/6/21 12:44 PM, Tony Battersby wrote: > On 8/6/21 1:33 AM, Richard Purdie wrote: >> On Thu, 2021-08-05 at 17:54 -0400, Tony Battersby wrote: >>> On 8/5/21 6:21 AM, Richard Purdie wrote: >>>> On Wed, 2021-08-04 at 20:13 -0400, Randy MacLeod wrote: >>>>> On 2021-07-27 9:34 a.m., Tony Battersby wrote: >>>>>> Currently, enabling Link Time Optimization via: >>>>>> >>>>>> require conf/distro/include/lto.inc >>>>>> DISTRO_FEATURES_append = " lto" >>>>>> >>>>>> makes the build non-reproducible (as in >>>>>> https://reproducible-builds.org/). This patch set fixes some of the >>>>>> problems, but is not yet a complete solution. >>>>>> >>>>>> This patch set addresses the following bugs: >>>>>> https://bugzilla.yoctoproject.org/show_bug.cgi?id=14481 >>>>>> https://bugzilla.yoctoproject.org/show_bug.cgi?id=14490 >>>>>> >>>>>> openembedded-core patches >>>>>> 1/4: lto.inc: disable LTO for grub >>>>>> 2/4: gcc: Backport patch to make LTO builds more reproducible >>>>>> 3/4: bitbake.conf: add DEBUG_PREFIX_MAP to TARGET_LDFLAGS >>>>>> >>>>>> openembedded-devel patches >>>>>> 4/4: net-snmp: fix QA Issue after LDFLAGS change >>>>>> >>>>>> Patch 3 adds DEBUG_PREFIX_MAP to TARGET_LDFLAGS (it was already in >>>>>> TARGET_CFLAGS). This could possibly be done differently, such as by >>>>>> removing it from TARGET_CFLAGS and adding it to TOOLCHAIN_OPTIONS, >>>>>> TARGET_CC_ARCH, or TUNE_CCARGS, but I haven't tested those. Let me know >>>>>> if there is a better/preferred way to do it. >>>>>> >>>>>> The TARGET_LDFLAGS change broke net-snmp from meta-networking, so I have >>>>>> included a patch to fix that. Other packages might have been broken as >>>>>> well, although I haven't found any others. >>>>>> >>>>>> With all these patches, there are still many shared libraries in /lib >>>>>> and /usr/lib that are still not reproducible with LTO enabled because >>>>>> x86_64-poky-linux-libtool filters out the -f*-prefix-map flags added by >>>>>> patch 3 when linking shared libraries. /usr/lib/libbz2.so.1.0.6 from >>>>>> the libbz2 package from the bzip2 recipe is a good example. I do not >>>>>> have a patch for that problem. I will leave that to someone else. >>>>>> >>>>>> So this is what we need to achieve reproducible LTO builds: >>>>>> *) This patch set >>>>>> *) A fix for libtool >>>>>> *) Add LTO to reproducible builds testing if appropriate (e.g. >>>>>> meta/lib/oeqa/selftest/cases/reproducible.py, CI on Autobuilder) >>>>>> *) Fixes for individual packages, if needed >>>>>> *) Once everything works, update the documentation again (I previously >>>>>> requested a documentation change for this bug, which will eventually >>>>>> show up at >>>>>> http://docs.yoctoproject.org/test-manual/reproducible-builds.html). >>>>> This is certainly a few steps in the right direction with a good outline >>>>> of what's next. >>>>> >>>>> I suspect that Richard has been busy with other things like overrides >>>>> and the update to glibc-2.34. >>>>> >>>>> Do you have any suggestions or questions about what we should do >>>>> with this patch series Richard? >>>> Thanks for the reminder, I've been meaning to queue/test this which I will >>>> do. The gcc version has been bumped but I ported the patch forward to it >>>> and the patches are in master-next for testing. It is great to see this >>>> support moving forward. >>>> >>>> Much as it pains me to admit having any libtool knowledge, I may be able to >>>> help with that bit. There is an untested patch below which may address that >>>> issue, or at least give a good hint on how to do it if it doesn't work. I >>>> don't have an environment to test easily right now. >>>> >>>> Cheers, >>>> >>>> Richard >>> I tested your libtool patch in master-next with the bzip2 recipe. It >>> works as intended; libbz2.so.1.0.6 and all the other output files are >>> now reproducible with LTO enabled. I also verified that libtool is >>> passing the -f*-prefix-map flags to the link command in log.do_compile >>> as intended. >>> >>> I also verified that core-image-minimal builds successfully with the >>> libtool patch. >> Thanks for testing that, sounds like it helps that issue. I'll queue that >> patch. >> >> Unfortunately even with the ruby fix I queued, we still see reproduciblity >> issues with the LDFLAGS change in curl-dev, gettext-ptest, ruby and ruby-dbg >> packages: >> >> https://autobuilder.yoctoproject.org/typhoon/#/builders/115/builds/515/steps/12/logs/stdio >> >> We'll need to resolve those before I can merge the flags change. >> >> Diffoscope output: >> >> https://autobuilder.yocto.io/pub/repro-fail/oe-reproducible-20210805-8jfd55ki/packages/diff-html/ >> >> Basically LDFLAGS get encoded into a couple of scripts and the ruby binary. >> >> Cheers, >> >> Richard >> > It looks like the ruby problem is being caused by MJIT_DLDFLAGS in > build/tmp/work/core2-64-poky-linux/ruby/3.0.2-r0/build/mjit_config.h. > There is already a patch > poky/meta/recipes-devtools/ruby/ruby/0001-template-Makefile.in-do-not-write-host-cross-cc-item.patch > that removes some of these items for the sake of reproducibiltiy. But I > do not know ruby, so I do not know the implications of doing the same > thing for MJIT_DLDFLAGS. > > Tony Battersby > > > Richard,
I noticed that you already fixed the problems in curl and gettext in master-next. Here is a patch for ruby if you haven't fixed that already. Tony Battersby >From 9cfb9777ff6ef95a27ed6a03719c807365d0f116 Mon Sep 17 00:00:00 2001 From: Tony Battersby <[email protected]> Date: Fri, 6 Aug 2021 16:47:14 -0400 Subject: [PATCH] ruby: Fix reproducibility issue with LDFLAGS If we add DEBUG_PREFIX_MAP into LDFLAGS, ruby and ruby-dbg are no longer reproducible. Fix this. Signed-off-by: Tony Battersby <[email protected]> --- ...-Makefile.in-filter-out-f-prefix-map.patch | 42 +++++++++++++++++++ meta/recipes-devtools/ruby/ruby_3.0.2.bb | 1 + 2 files changed, 43 insertions(+) create mode 100644 meta/recipes-devtools/ruby/ruby/0002-template-Makefile.in-filter-out-f-prefix-map.patch diff --git a/meta/recipes-devtools/ruby/ruby/0002-template-Makefile.in-filter-out-f-prefix-map.patch b/meta/recipes-devtools/ruby/ruby/0002-template-Makefile.in-filter-out-f-prefix-map.patch new file mode 100644 index 0000000000..9387506c26 --- /dev/null +++ b/meta/recipes-devtools/ruby/ruby/0002-template-Makefile.in-filter-out-f-prefix-map.patch @@ -0,0 +1,42 @@ +Subject: [PATCH] template/Makefile.in: filter out -f*prefix-map + +If we add DEBUG_PREFIX_MAP into LDFLAGS, ruby and ruby-dbg are no longer +reproducible. Fix this. + +Upstream-Status: Inapproppriate [oe-core specific] +Signed-off-by: Tony Battersby <[email protected]> +--- +--- a/tool/mjit_archflag.sh ++++ b/tool/mjit_archflag.sh +@@ -7,6 +7,20 @@ quote() { + echo + } + ++quote_filtered() { ++ printf "#${indent}define $1" ++ while shift && [ "$#" -gt 0 ]; do ++ case "$1" in ++ -ffile-prefix-map=*|-fdebug-prefix-map=*|-fmacro-prefix-map=*) ++ ;; ++ *) ++ printf ' "%s"'$sep "$1" ++ ;; ++ esac ++ done ++ echo ++} ++ + archs="" + arch_flag="" + +--- a/template/Makefile.in ++++ b/template/Makefile.in +@@ -666,7 +666,7 @@ mjit_config.h: + quote "MJIT_OPTFLAGS " $(MJIT_OPTFLAGS); \ + quote "MJIT_DEBUGFLAGS " $(MJIT_DEBUGFLAGS); \ + quote "MJIT_LDSHARED " ; \ +- quote "MJIT_DLDFLAGS MJIT_ARCHFLAG" $(MJIT_DLDFLAGS); \ ++ quote_filtered "MJIT_DLDFLAGS MJIT_ARCHFLAG" $(MJIT_DLDFLAGS); \ + quote "MJIT_LIBS " $(LIBRUBYARG_SHARED); \ + quote 'PRELOADENV "@PRELOADENV@"'; \ + indent=$${archs:+' '}; \ diff --git a/meta/recipes-devtools/ruby/ruby_3.0.2.bb b/meta/recipes-devtools/ruby/ruby_3.0.2.bb index 0b9c342def..38e594a59e 100644 --- a/meta/recipes-devtools/ruby/ruby_3.0.2.bb +++ b/meta/recipes-devtools/ruby/ruby_3.0.2.bb @@ -6,6 +6,7 @@ SRC_URI += " \ file://remove_has_include_macros.patch \ file://run-ptest \ file://0001-template-Makefile.in-do-not-write-host-cross-cc-item.patch \ + file://0002-template-Makefile.in-filter-out-f-prefix-map.patch \ " SRC_URI[sha256sum] = "5085dee0ad9f06996a8acec7ebea4a8735e6fac22f22e2d98c3f2bc3bef7e6f1" -- 2.25.1
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#154575): https://lists.openembedded.org/g/openembedded-core/message/154575 Mute This Topic: https://lists.openembedded.org/mt/84481652/21656 Group Owner: [email protected] Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]] -=-=-=-=-=-=-=-=-=-=-=-
