I am seeing bunch of failures on meta-oe and meta-atmel https://errors.yoctoproject.org/Errors/Details/600064/ https://errors.yoctoproject.org/Errors/Details/600065/ https://errors.yoctoproject.org/Errors/Details/600071/
On Fri, Aug 6, 2021 at 2:04 PM Tony Battersby <to...@cybernetics.com> wrote: > > On 8/6/21 12:44 PM, Tony Battersby wrote: > > On 8/6/21 1:33 AM, Richard Purdie wrote: > >> On Thu, 2021-08-05 at 17:54 -0400, Tony Battersby wrote: > >>> On 8/5/21 6:21 AM, Richard Purdie wrote: > >>>> On Wed, 2021-08-04 at 20:13 -0400, Randy MacLeod wrote: > >>>>> On 2021-07-27 9:34 a.m., Tony Battersby wrote: > >>>>>> Currently, enabling Link Time Optimization via: > >>>>>> > >>>>>> require conf/distro/include/lto.inc > >>>>>> DISTRO_FEATURES_append = " lto" > >>>>>> > >>>>>> makes the build non-reproducible (as in > >>>>>> https://reproducible-builds.org/). This patch set fixes some of the > >>>>>> problems, but is not yet a complete solution. > >>>>>> > >>>>>> This patch set addresses the following bugs: > >>>>>> https://bugzilla.yoctoproject.org/show_bug.cgi?id=14481 > >>>>>> https://bugzilla.yoctoproject.org/show_bug.cgi?id=14490 > >>>>>> > >>>>>> openembedded-core patches > >>>>>> 1/4: lto.inc: disable LTO for grub > >>>>>> 2/4: gcc: Backport patch to make LTO builds more reproducible > >>>>>> 3/4: bitbake.conf: add DEBUG_PREFIX_MAP to TARGET_LDFLAGS > >>>>>> > >>>>>> openembedded-devel patches > >>>>>> 4/4: net-snmp: fix QA Issue after LDFLAGS change > >>>>>> > >>>>>> Patch 3 adds DEBUG_PREFIX_MAP to TARGET_LDFLAGS (it was already in > >>>>>> TARGET_CFLAGS). This could possibly be done differently, such as by > >>>>>> removing it from TARGET_CFLAGS and adding it to TOOLCHAIN_OPTIONS, > >>>>>> TARGET_CC_ARCH, or TUNE_CCARGS, but I haven't tested those. Let me > >>>>>> know > >>>>>> if there is a better/preferred way to do it. > >>>>>> > >>>>>> The TARGET_LDFLAGS change broke net-snmp from meta-networking, so I > >>>>>> have > >>>>>> included a patch to fix that. Other packages might have been broken as > >>>>>> well, although I haven't found any others. > >>>>>> > >>>>>> With all these patches, there are still many shared libraries in /lib > >>>>>> and /usr/lib that are still not reproducible with LTO enabled because > >>>>>> x86_64-poky-linux-libtool filters out the -f*-prefix-map flags added by > >>>>>> patch 3 when linking shared libraries. /usr/lib/libbz2.so.1.0.6 from > >>>>>> the libbz2 package from the bzip2 recipe is a good example. I do not > >>>>>> have a patch for that problem. I will leave that to someone else. > >>>>>> > >>>>>> So this is what we need to achieve reproducible LTO builds: > >>>>>> *) This patch set > >>>>>> *) A fix for libtool > >>>>>> *) Add LTO to reproducible builds testing if appropriate (e.g. > >>>>>> meta/lib/oeqa/selftest/cases/reproducible.py, CI on Autobuilder) > >>>>>> *) Fixes for individual packages, if needed > >>>>>> *) Once everything works, update the documentation again (I previously > >>>>>> requested a documentation change for this bug, which will eventually > >>>>>> show up at > >>>>>> http://docs.yoctoproject.org/test-manual/reproducible-builds.html). > >>>>> This is certainly a few steps in the right direction with a good outline > >>>>> of what's next. > >>>>> > >>>>> I suspect that Richard has been busy with other things like overrides > >>>>> and the update to glibc-2.34. > >>>>> > >>>>> Do you have any suggestions or questions about what we should do > >>>>> with this patch series Richard? > >>>> Thanks for the reminder, I've been meaning to queue/test this which I > >>>> will > >>>> do. The gcc version has been bumped but I ported the patch forward to it > >>>> and the patches are in master-next for testing. It is great to see this > >>>> support moving forward. > >>>> > >>>> Much as it pains me to admit having any libtool knowledge, I may be able > >>>> to > >>>> help with that bit. There is an untested patch below which may address > >>>> that > >>>> issue, or at least give a good hint on how to do it if it doesn't work. I > >>>> don't have an environment to test easily right now. > >>>> > >>>> Cheers, > >>>> > >>>> Richard > >>> I tested your libtool patch in master-next with the bzip2 recipe. It > >>> works as intended; libbz2.so.1.0.6 and all the other output files are > >>> now reproducible with LTO enabled. I also verified that libtool is > >>> passing the -f*-prefix-map flags to the link command in log.do_compile > >>> as intended. > >>> > >>> I also verified that core-image-minimal builds successfully with the > >>> libtool patch. > >> Thanks for testing that, sounds like it helps that issue. I'll queue that > >> patch. > >> > >> Unfortunately even with the ruby fix I queued, we still see reproduciblity > >> issues with the LDFLAGS change in curl-dev, gettext-ptest, ruby and > >> ruby-dbg > >> packages: > >> > >> https://autobuilder.yoctoproject.org/typhoon/#/builders/115/builds/515/steps/12/logs/stdio > >> > >> We'll need to resolve those before I can merge the flags change. > >> > >> Diffoscope output: > >> > >> https://autobuilder.yocto.io/pub/repro-fail/oe-reproducible-20210805-8jfd55ki/packages/diff-html/ > >> > >> Basically LDFLAGS get encoded into a couple of scripts and the ruby binary. > >> > >> Cheers, > >> > >> Richard > >> > > It looks like the ruby problem is being caused by MJIT_DLDFLAGS in > > build/tmp/work/core2-64-poky-linux/ruby/3.0.2-r0/build/mjit_config.h. > > There is already a patch > > poky/meta/recipes-devtools/ruby/ruby/0001-template-Makefile.in-do-not-write-host-cross-cc-item.patch > > that removes some of these items for the sake of reproducibiltiy. But I > > do not know ruby, so I do not know the implications of doing the same > > thing for MJIT_DLDFLAGS. > > > > Tony Battersby > > > > > > > Richard, > > I noticed that you already fixed the problems in curl and gettext in > master-next. Here is a patch for ruby if you haven't fixed that already. > > Tony Battersby > > From 9cfb9777ff6ef95a27ed6a03719c807365d0f116 Mon Sep 17 00:00:00 2001 > From: Tony Battersby <to...@cybernetics.com> > Date: Fri, 6 Aug 2021 16:47:14 -0400 > Subject: [PATCH] ruby: Fix reproducibility issue with LDFLAGS > > If we add DEBUG_PREFIX_MAP into LDFLAGS, ruby and ruby-dbg are no > longer reproducible. Fix this. > > Signed-off-by: Tony Battersby <to...@cybernetics.com> > --- > ...-Makefile.in-filter-out-f-prefix-map.patch | 42 +++++++++++++++++++ > meta/recipes-devtools/ruby/ruby_3.0.2.bb | 1 + > 2 files changed, 43 insertions(+) > create mode 100644 > meta/recipes-devtools/ruby/ruby/0002-template-Makefile.in-filter-out-f-prefix-map.patch > > diff --git > a/meta/recipes-devtools/ruby/ruby/0002-template-Makefile.in-filter-out-f-prefix-map.patch > > b/meta/recipes-devtools/ruby/ruby/0002-template-Makefile.in-filter-out-f-prefix-map.patch > new file mode 100644 > index 0000000000..9387506c26 > --- /dev/null > +++ > b/meta/recipes-devtools/ruby/ruby/0002-template-Makefile.in-filter-out-f-prefix-map.patch > @@ -0,0 +1,42 @@ > +Subject: [PATCH] template/Makefile.in: filter out -f*prefix-map > + > +If we add DEBUG_PREFIX_MAP into LDFLAGS, ruby and ruby-dbg are no longer > +reproducible. Fix this. > + > +Upstream-Status: Inapproppriate [oe-core specific] > +Signed-off-by: Tony Battersby <to...@cybernetics.com> > +--- > +--- a/tool/mjit_archflag.sh > ++++ b/tool/mjit_archflag.sh > +@@ -7,6 +7,20 @@ quote() { > + echo > + } > + > ++quote_filtered() { > ++ printf "#${indent}define $1" > ++ while shift && [ "$#" -gt 0 ]; do > ++ case "$1" in > ++ -ffile-prefix-map=*|-fdebug-prefix-map=*|-fmacro-prefix-map=*) > ++ ;; > ++ *) > ++ printf ' "%s"'$sep "$1" > ++ ;; > ++ esac > ++ done > ++ echo > ++} > ++ > + archs="" > + arch_flag="" > + > +--- a/template/Makefile.in > ++++ b/template/Makefile.in > +@@ -666,7 +666,7 @@ mjit_config.h: > + quote "MJIT_OPTFLAGS " $(MJIT_OPTFLAGS); \ > + quote "MJIT_DEBUGFLAGS " $(MJIT_DEBUGFLAGS); \ > + quote "MJIT_LDSHARED " ; \ > +- quote "MJIT_DLDFLAGS MJIT_ARCHFLAG" $(MJIT_DLDFLAGS); \ > ++ quote_filtered "MJIT_DLDFLAGS MJIT_ARCHFLAG" $(MJIT_DLDFLAGS); \ > + quote "MJIT_LIBS " $(LIBRUBYARG_SHARED); \ > + quote 'PRELOADENV "@PRELOADENV@"'; \ > + indent=$${archs:+' '}; \ > diff --git a/meta/recipes-devtools/ruby/ruby_3.0.2.bb > b/meta/recipes-devtools/ruby/ruby_3.0.2.bb > index 0b9c342def..38e594a59e 100644 > --- a/meta/recipes-devtools/ruby/ruby_3.0.2.bb > +++ b/meta/recipes-devtools/ruby/ruby_3.0.2.bb > @@ -6,6 +6,7 @@ SRC_URI += " \ > file://remove_has_include_macros.patch \ > file://run-ptest \ > > file://0001-template-Makefile.in-do-not-write-host-cross-cc-item.patch \ > + file://0002-template-Makefile.in-filter-out-f-prefix-map.patch \ > " > > SRC_URI[sha256sum] = > "5085dee0ad9f06996a8acec7ebea4a8735e6fac22f22e2d98c3f2bc3bef7e6f1" > -- > 2.25.1 >
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#154578): https://lists.openembedded.org/g/openembedded-core/message/154578 Mute This Topic: https://lists.openembedded.org/mt/84481652/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
