Re: [PATCH/RFC] kernel-defaults.mk: get rid of BuildID
Hi, > On 5. Apr 2022, at 21:33, Bjørn Mork wrote: > > Daniel Golle writes: > >> You probably meant LDFLAGS_vmlinux because from what I understand >> KBUILD_LDFLAGS_MODULE only applies when building modules but not when >> linking vmlinux. >> As ld only cares about the last mentioned --build-id= parameter >> supplied, we can override it using KBUILD_LDFLAGS (which should apply >> to both, vmlinux.elf as well as modules). >> I haven't tried any of that yet though. > > How about simply making this configurable upstream? If more than one > distro needs it, then I can't imagine it should be a problem to get have > it accepted. Possibly with a "depends on EXPERT" or similar. > I created a patch which we could test downstream: https://github.com/openwrt/openwrt/pull/9669 Since it’s used by friends over at Debian too, we might want to get it upstream. ___ openwrt-devel mailing list openwrt-devel@lists.openwrt.org https://lists.openwrt.org/mailman/listinfo/openwrt-devel
Re: [PATCH/RFC] kernel-defaults.mk: get rid of BuildID
Daniel Golle writes: > You probably meant LDFLAGS_vmlinux because from what I understand > KBUILD_LDFLAGS_MODULE only applies when building modules but not when > linking vmlinux. > As ld only cares about the last mentioned --build-id= parameter > supplied, we can override it using KBUILD_LDFLAGS (which should apply > to both, vmlinux.elf as well as modules). > I haven't tried any of that yet though. How about simply making this configurable upstream? If more than one distro needs it, then I can't imagine it should be a problem to get have it accepted. Possibly with a "depends on EXPERT" or similar. Bjørn ___ openwrt-devel mailing list openwrt-devel@lists.openwrt.org https://lists.openwrt.org/mailman/listinfo/openwrt-devel
Re: [PATCH/RFC] kernel-defaults.mk: get rid of BuildID
On 05.04.22 20:51, Daniel Golle wrote: On Tue, Apr 05, 2022 at 05:05:43PM +0200, Felix Fietkau wrote: On 05.04.22 03:14, Daniel Golle wrote: > When building the Linux kernel, the linker generates a hash of all > versions of tools involved in a build called BuildID in ELF header. > This breaks reproducibility accross different buildhosts eventhough > OpenWrt builds the toolchain from source -- the build-id hash ends up > to be the only thing which differs in the resulting builds. > > The cause is most likely a result of the build hosts' architectures, > OSs and standard C libraries being different. > > While in theory it is true that tools may produce a different output > depending on archtecture, OS and libc of the buildhost, in practice > this is (fortunately) hardly ever the case and hence it contradicts > ld(1) which states: > > 'The "md5" and "sha1" styles produces an identifier that is always >the same in an identical output file, but will be unique among all >nonidentical output files.' > > (the kernel is using sha1 style build-id, rebuilding the kernel on a > different buildhost results in everything being identical **except** > for the build-id) > > Hence, to achieve reproducible builds we will either have to resort to > identical containers/VMs for building or get rid of the BuildID hash > alltogether (or use a different build-id style) > > At this point, this seems to be what Debian is doing as well in order > to achieve reproducible kernel builds, see[1]. > > [1]: https://wiki.debian.org/SameKernel#How_this_works > Signed-off-by: Daniel Golle > > diff --git a/include/kernel-defaults.mk b/include/kernel-defaults.mk > index 1e82f7d739..9c8d5fbe97 100644 > --- a/include/kernel-defaults.mk > +++ b/include/kernel-defaults.mk > @@ -46,6 +46,7 @@ else >if [ -d $(LINUX_DIR)/user_headers ]; then \ >rm -rf $(LINUX_DIR)/user_headers; \ >fi > + $(SED) -i $(LINUX_DIR)/Makefile -e 's/--build-id=.*/--build-id=none/g' I don't like running sed on the linux Makefile, as this interferes with creating patches for it. I think it would be better to simply override KBUILD_LDFLAGS_MODULE on the kernel/module build command line. You probably meant LDFLAGS_vmlinux because from what I understand KBUILD_LDFLAGS_MODULE only applies when building modules but not when linking vmlinux. As ld only cares about the last mentioned --build-id= parameter supplied, we can override it using KBUILD_LDFLAGS (which should apply to both, vmlinux.elf as well as modules). I haven't tried any of that yet though.Right, I overlooked that one. Either way, you likely need to patch the kernel in order to not have to override the full set of linker arguments. I still think explicit patching + variable override is preferable over sed based Makefile patching. - Felix ___ openwrt-devel mailing list openwrt-devel@lists.openwrt.org https://lists.openwrt.org/mailman/listinfo/openwrt-devel
Re: [PATCH/RFC] kernel-defaults.mk: get rid of BuildID
On Tue, Apr 05, 2022 at 05:05:43PM +0200, Felix Fietkau wrote: > On 05.04.22 03:14, Daniel Golle wrote: > > When building the Linux kernel, the linker generates a hash of all > > versions of tools involved in a build called BuildID in ELF header. > > This breaks reproducibility accross different buildhosts eventhough > > OpenWrt builds the toolchain from source -- the build-id hash ends up > > to be the only thing which differs in the resulting builds. > > > > The cause is most likely a result of the build hosts' architectures, > > OSs and standard C libraries being different. > > > > While in theory it is true that tools may produce a different output > > depending on archtecture, OS and libc of the buildhost, in practice > > this is (fortunately) hardly ever the case and hence it contradicts > > ld(1) which states: > > > > 'The "md5" and "sha1" styles produces an identifier that is always > >the same in an identical output file, but will be unique among all > >nonidentical output files.' > > > > (the kernel is using sha1 style build-id, rebuilding the kernel on a > > different buildhost results in everything being identical **except** > > for the build-id) > > > > Hence, to achieve reproducible builds we will either have to resort to > > identical containers/VMs for building or get rid of the BuildID hash > > alltogether (or use a different build-id style) > > > > At this point, this seems to be what Debian is doing as well in order > > to achieve reproducible kernel builds, see[1]. > > > > [1]: https://wiki.debian.org/SameKernel#How_this_works > > Signed-off-by: Daniel Golle > > > > diff --git a/include/kernel-defaults.mk b/include/kernel-defaults.mk > > index 1e82f7d739..9c8d5fbe97 100644 > > --- a/include/kernel-defaults.mk > > +++ b/include/kernel-defaults.mk > > @@ -46,6 +46,7 @@ else > > if [ -d $(LINUX_DIR)/user_headers ]; then \ > > rm -rf $(LINUX_DIR)/user_headers; \ > > fi > > + $(SED) -i $(LINUX_DIR)/Makefile -e 's/--build-id=.*/--build-id=none/g' > I don't like running sed on the linux Makefile, as this interferes with > creating patches for it. I think it would be better to simply override > KBUILD_LDFLAGS_MODULE on the kernel/module build command line. You probably meant LDFLAGS_vmlinux because from what I understand KBUILD_LDFLAGS_MODULE only applies when building modules but not when linking vmlinux. As ld only cares about the last mentioned --build-id= parameter supplied, we can override it using KBUILD_LDFLAGS (which should apply to both, vmlinux.elf as well as modules). I haven't tried any of that yet though. ___ openwrt-devel mailing list openwrt-devel@lists.openwrt.org https://lists.openwrt.org/mailman/listinfo/openwrt-devel
Re: [PATCH/RFC] kernel-defaults.mk: get rid of BuildID
please forgive me stupidity, I couldn't understand the last part of your recommendation: Daniel Golle wrote: > Hence, to achieve reproducible builds we will either have to resort to > identical containers/VMs for building or get rid of the BuildID hash > alltogether (or use a different build-id style) A) identical containers/VMs B) get rid of BuildID C) use a different build-id style > At this point, this seems to be what Debian is doing as well in order > to achieve reproducible kernel builds, see[1]. And which is Debian doing? :-) > [1]: https://wiki.debian.org/SameKernel#How_this_works I read this page and I think it's temporarily (B), but unclear if they are going to (A). (I prefer a path that leads to the build-id meaning that the same versions of the same compilers on the same host OS were used, but it would be nice not to require the same compile of those compilers...) signature.asc Description: PGP signature ___ openwrt-devel mailing list openwrt-devel@lists.openwrt.org https://lists.openwrt.org/mailman/listinfo/openwrt-devel
Re: [PATCH/RFC] kernel-defaults.mk: get rid of BuildID
On 05.04.22 03:14, Daniel Golle wrote: When building the Linux kernel, the linker generates a hash of all versions of tools involved in a build called BuildID in ELF header. This breaks reproducibility accross different buildhosts eventhough OpenWrt builds the toolchain from source -- the build-id hash ends up to be the only thing which differs in the resulting builds. The cause is most likely a result of the build hosts' architectures, OSs and standard C libraries being different. While in theory it is true that tools may produce a different output depending on archtecture, OS and libc of the buildhost, in practice this is (fortunately) hardly ever the case and hence it contradicts ld(1) which states: 'The "md5" and "sha1" styles produces an identifier that is always the same in an identical output file, but will be unique among all nonidentical output files.' (the kernel is using sha1 style build-id, rebuilding the kernel on a different buildhost results in everything being identical **except** for the build-id) Hence, to achieve reproducible builds we will either have to resort to identical containers/VMs for building or get rid of the BuildID hash alltogether (or use a different build-id style) At this point, this seems to be what Debian is doing as well in order to achieve reproducible kernel builds, see[1]. [1]: https://wiki.debian.org/SameKernel#How_this_works Signed-off-by: Daniel Golle diff --git a/include/kernel-defaults.mk b/include/kernel-defaults.mk index 1e82f7d739..9c8d5fbe97 100644 --- a/include/kernel-defaults.mk +++ b/include/kernel-defaults.mk @@ -46,6 +46,7 @@ else if [ -d $(LINUX_DIR)/user_headers ]; then \ rm -rf $(LINUX_DIR)/user_headers; \ fi + $(SED) -i $(LINUX_DIR)/Makefile -e 's/--build-id=.*/--build-id=none/g' I don't like running sed on the linux Makefile, as this interferes with creating patches for it. I think it would be better to simply override KBUILD_LDFLAGS_MODULE on the kernel/module build command line. - Felix ___ openwrt-devel mailing list openwrt-devel@lists.openwrt.org https://lists.openwrt.org/mailman/listinfo/openwrt-devel
Re: [PATCH/RFC] kernel-defaults.mk: get rid of BuildID
Hi, >> +$(SED) -i $(LINUX_DIR)/Makefile -e >> 's/--build-id=.*/--build-id=0x$(LINUX_VERMAGIC)/g’ This doesn’t fly since LINUX_VERMAGIC (based on .vermagic) is based on the Kernel configuration and only available after the Configuration step. I moved it from the Prepare to the end of Configuration and it works fine: diff --git a/include/kernel-defaults.mk b/include/kernel-defaults.mk index 1e82f7d739..63da2ea038 100644 --- a/include/kernel-defaults.mk +++ b/include/kernel-defaults.mk @@ -119,6 +119,7 @@ define Kernel/Configure/Default } $(_SINGLE) [ -d $(LINUX_DIR)/user_headers ] || $(KERNEL_MAKE) INSTALL_HDR_PATH=$(LINUX_DIR)/user_headers headers_install grep '=[ym]' $(LINUX_DIR)/.config.set | LC_ALL=C sort | $(MKHASH) md5 > $(LINUX_DIR)/.vermagic + $(SED) "s/--build-id=.*/--build-id=0x(cat $(LINUX_DIR)/.vermagic)/g" $(LINUX_DIR)/Makefile endef It works as expected: ubuntu@primary:~/a$ cat /home/ubuntu/a/build_dir/target-x86_64_musl/linux-x86_64/linux-5.10.109/Makefile | grep build-id KBUILD_LDFLAGS_MODULE += --build-id=0xc096cf71d9bd3c319494033a0e38394b ubuntu@primary:~$ file kernela kernela: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, BuildID[md5/uuid]=c096cf71d9bd3c319494033a0e38394b, stripped ubuntu@primary:~/a$ make -C target/linux val.LINUX_VERMAGIC make: Entering directory '/home/ubuntu/a/target/linux' c096cf71d9bd3c319494033a0e38394b make: Leaving directory '/home/ubuntu/a/target/linux’ Best, Paul ___ openwrt-devel mailing list openwrt-devel@lists.openwrt.org https://lists.openwrt.org/mailman/listinfo/openwrt-devel
Re: [PATCH/RFC] kernel-defaults.mk: get rid of BuildID
Hi, > To investigate the issue of non-reproducible kernel images accross > buildhosts I compated the files build by OpenWrt's buildbots with > Paul's rebuilder script running on his (Aarch64) Mac. Sorry there was a misunderstanding. I’m building on macOS to find extra issues but the “rebuild” files I sent you earlier are created by the GitHub CI using Ubuntu on x86[1]. [1]: https://github.com/aparcar/openwrt-rebuilder/actions/runs/2089052638 > + $(SED) -i $(LINUX_DIR)/Makefile -e > 's/--build-id=.*/--build-id=0x$(LINUX_VERMAGIC)/g’ Looks good, I’ll try this. Paul ___ openwrt-devel mailing list openwrt-devel@lists.openwrt.org https://lists.openwrt.org/mailman/listinfo/openwrt-devel
Re: [PATCH/RFC] kernel-defaults.mk: get rid of BuildID
Daniel Golle [2022-04-05 02:14:09]:
Hi,
thanks a lot for your and Paul's reproducible efforts!
> diff --git a/include/kernel-defaults.mk b/include/kernel-defaults.mk
> index 1e82f7d739..9c8d5fbe97 100644
> --- a/include/kernel-defaults.mk
> +++ b/include/kernel-defaults.mk
> @@ -46,6 +46,7 @@ else
> if [ -d $(LINUX_DIR)/user_headers ]; then \
> rm -rf $(LINUX_DIR)/user_headers; \
> fi
BTW we likely have LINUX_VERMAGIC md5 hash generated over kernel config
symbols:
grep '=[ym]' $(LINUX_DIR)/.config.set | LC_ALL=C sort | $(MKHASH) md5 >
$(LINUX_DIR)/.vermagic
LINUX_VERMAGIC:=$(strip $(shell cat $(LINUX_DIR)/.vermagic 2>/dev/null))
So it makes me wonder if we could use something like this instead (untested):
> + $(SED) -i $(LINUX_DIR)/Makefile -e 's/--build-id=.*/--build-id=none/g'
+ $(SED) -i $(LINUX_DIR)/Makefile -e
's/--build-id=.*/--build-id=0x$(LINUX_VERMAGIC)/g'
>From ld(1) `--build-id=style` help:
or "0x hexstring " to use a chosen bit string specified as an even number of
hexadecimal digits ("-" and ":" characters between digit pairs are ignored).
Having some kind of build ID is sometimes handy, for example during
troubleshooting.
Cheers,
Petr
___
openwrt-devel mailing list
openwrt-devel@lists.openwrt.org
https://lists.openwrt.org/mailman/listinfo/openwrt-devel
