Re: [Reproducible-builds] [U-Boot] [PATCH] build: create time and date independent binary
Hello Vagrant, Am 19.07.2015 um 15:14 schrieb Vagrant Cascadian: On 2015-07-19, Holger Levsen wrote: All this said, if you send me patches, I will probably deploy them as I'm very curious and more reproducibility efforts are good :-) We can can always decide to remove or move them later. I wish to make all contributions upstream. What would really help at first would be to have all targets built regularly to see where work is needed. This is where I think the Debian infrastructure could help, in a similar way as what was started for Coreboot. FWIW, I was planning on including this patch to u-boot in the next upload to Debian: https://anonscm.debian.org/cgit/collab-maint/u-boot.git/tree/debian/patches/use-date-from-debian-changelog.patch?h=experimental-2015.07 I *think* that actually makes u-boot build reproducibly with Debian's reproducible builds toolchain when SOURCE_DATE_EPOCH is set, but I haven't tested it fully. I might have missed some other sources of non-determinism... Hoping to get some armhf buildd nodes up an running soonish... although it should also be buildable with the cross-toolchains, if the reproducible buildds coulld be made to support that. Did you see my v2 patch [1] for u-boot? It also builds now u-boot images created with mkimage reproducible ... If I interpret your patch from above correct, you add with SOURCE_DATE_EPOCH a specific fix timestamp? I think, this could be included to my approach too ... If SOURCE_DATE_EPOCH is defined, use it, fi not defined define U_BOOT_DATE, U_BOOT_TIME and U_BOOT_TZ with my default settings from [1] ... All only if CONFIG_SYS_EXACT_BINARY is set in the u-boot config of course ... What do you think? bye, Heiko [1] http://patchwork.ozlabs.org/patch/487530/ -- DENX Software Engineering GmbH, Managing Director: Wolfgang Denk HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: [Reproducible-builds] [U-Boot] [PATCH] build: create time and date independent binary
Hi Paul, sorry for the late reply. On Samstag, 13. Juni 2015, Paul Kocialkowski wrote: you've seen https://reproducible.debian.net/u-boot ? This seems very minimalistic, but it's good to see U-Boot was given some attention already! :-) but maybe you can explain why u-boot needs more reproducibility testing than what there currently is. i'm definitly interested and not opposed, even though I think there shoukd be good reasons to treat some software specially. The point is to make U-Boot reproducible for all possible targets, not only the few ones that are supported by U-Boot. I think your sentence is missing some word…?!? I think this requires some extra infrastructure. In that sense, it is very similar to Coreboot. (also please note that we currently only have amd64 hw to run our tests on.) The problem is the same as Coreboot, which uses its own toolchain to build images. We don't need to have native armhf builds for U-Boot, testing with the armhf toolchain that is in Debian should be enough. I see. I understand, this works out nicely because all the work on Coreboot will be inherited by Libreboot. However, on U-Boot, the work to bring reproducible builds has to take place initially. I know for a fact that parts of the code use things like __FILE__ or timestamps. Ah. That makes sense. For U-Boot, it will certainly make sense for the distributions packaging it. I'm the main developer of Replicant, the fully free version of Android, and it would definitely be useful to have a smartphone on which we can trust that the bootloader can be built in a reproducible manner (and checked after being installed). indeed! All this said, if you send me patches, I will probably deploy them as I'm very curious and more reproducibility efforts are good :-) We can can always decide to remove or move them later. I wish to make all contributions upstream. What would really help at first would be to have all targets built regularly to see where work is needed. This is where I think the Debian infrastructure could help, in a similar way as what was started for Coreboot. can you point me to a how to explaining this or tell me those steps, starting with git clone...? cheers, Holger signature.asc Description: This is a digitally signed message part. ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: [Reproducible-builds] [U-Boot] [PATCH] build: create time and date independent binary
On 2015-07-19, Holger Levsen wrote: All this said, if you send me patches, I will probably deploy them as I'm very curious and more reproducibility efforts are good :-) We can can always decide to remove or move them later. I wish to make all contributions upstream. What would really help at first would be to have all targets built regularly to see where work is needed. This is where I think the Debian infrastructure could help, in a similar way as what was started for Coreboot. FWIW, I was planning on including this patch to u-boot in the next upload to Debian: https://anonscm.debian.org/cgit/collab-maint/u-boot.git/tree/debian/patches/use-date-from-debian-changelog.patch?h=experimental-2015.07 I *think* that actually makes u-boot build reproducibly with Debian's reproducible builds toolchain when SOURCE_DATE_EPOCH is set, but I haven't tested it fully. I might have missed some other sources of non-determinism... Hoping to get some armhf buildd nodes up an running soonish... although it should also be buildable with the cross-toolchains, if the reproducible buildds coulld be made to support that. live well, vagrant signature.asc Description: PGP signature ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: [Reproducible-builds] [U-Boot] [PATCH] build: create time and date independent binary
Hi Paul, On Sonntag, 19. Juli 2015, Paul Kocialkowski wrote: sorry for the late reply. No problem, I haven't kept you or the list posted on this either. I got a chance to discuss the issue with Lunar during RMLL 2015 and we came up with a nice way of doing things, using SOURCE_DATE_EPOCH. oh, nice! you've seen https://reproducible.debian.net/u-boot ? As far as I understood, this was only for the U-Boot tools. yes There seem to be two solutions to this: * Including a script (e.g. the one from coreboot) to build the toolchain as part of the build process * Using native builds with visualization on the servers that run the builds for the reproducible task force I tend to prefer the second one since it only requires a one-time setup cost, while the other one, that requires to build toolchains for each test build, implies a considerably longer build time for each test. these hosts also need maintenance so I actually prefer the first. The basics for building U-Boot are the following (e.g. for the Cubieboard2 target) git clone git://git.denx.de/u-boot.git cd u-boot make -C $SRC O=$DST CROSS_COMPILE=$CROSS_COMPILE Cubieboard2_defconfig make -C $SRC O=$DST CROSS_COMPILE=$CROSS_COMPILE aint there a makefile with a proper target? Also how to build these cross compilers? Let me know if you need more indications on this. yes, please. best a complete runnable script, aka a _working_ 5-10 liner, not the fancy one with variations, rebuilds and running debbindiff. Just the above 4 lines in working - (those 4 lines dont work cause the variables are not defined...) cheers, Holger signature.asc Description: This is a digitally signed message part. ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: [Reproducible-builds] [U-Boot] [PATCH] build: create time and date independent binary
Le dimanche 19 juillet 2015 à 17:47 +0200, Holger Levsen a écrit : There seem to be two solutions to this: * Including a script (e.g. the one from coreboot) to build the toolchain as part of the build process * Using native builds with visualization on the servers that run the builds for the reproducible task force I tend to prefer the second one since it only requires a one-time setup cost, while the other one, that requires to build toolchains for each test build, implies a considerably longer build time for each test. these hosts also need maintenance so I actually prefer the first. I understand. We could raise the topic on the U-Boot mailing list and see whether adding the scripts to build the cross-compilers tu U-Boot would be doable or not. Otherwise, we can just grab those scripts aside the U-Boot source code for the reproducible task, so that we don't have to wait for a long discussion to conclude. The basics for building U-Boot are the following (e.g. for the Cubieboard2 target) git clone git://git.denx.de/u-boot.git cd u-boot make -C $SRC O=$DST CROSS_COMPILE=$CROSS_COMPILE Cubieboard2_defconfig make -C $SRC O=$DST CROSS_COMPILE=$CROSS_COMPILE aint there a makefile with a proper target? Also how to build these cross compilers? I am using those scripts, that I wrote: http://git.paulk.fr/gitweb/?p=embedded-freedom-scripts.git;a=tree;f=u-boot;h=f735bc015b056c3b63a0144703d7f7168382ca59;hb=HEAD That can help but is not really usable as-is. Let me know if you need more indications on this. yes, please. best a complete runnable script, aka a _working_ 5-10 liner, not the fancy one with variations, rebuilds and running debbindiff. Just the above 4 lines in working - (those 4 lines dont work cause the variables are not defined...) I could do that, but we have to figure out the toolchain issue first, I suppose. -- Paul Kocialkowski, Replicant developer Replicant is a fully free Android distribution running on several devices, a free software mobile operating system putting the emphasis on freedom and privacy/security. Website: http://www.replicant.us/ Blog: http://blog.replicant.us/ Wiki/tracker/forums: http://redmine.replicant.us/ signature.asc Description: This is a digitally signed message part ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: [Reproducible-builds] [U-Boot] [PATCH] build: create time and date independent binary
Hi, On Sonntag, 19. Juli 2015, Paul Kocialkowski wrote: I *think* that actually makes u-boot build reproducibly with Debian's reproducible builds toolchain when SOURCE_DATE_EPOCH is set, but I haven't tested it fully. I might have missed some other sources of non-determinism... We came up with something similar last week. This is not actually sufficient, but close. I will submit what I have to the U-Boot mailing list soon (hopefully, later today), feel free to review it there and import it to Debian. cool! Native builds don't seem realistic given the many different architectures that U-Boot supports, unless done with virtualisation (see my other email from today). [...] Either way, we can always try to add a toolchain build script to U-Boot. I agree, I also think we should test uboot with such a script, similar like we do for coreboot. That wouldnt need armhf buildds. That said, making use of these armhf buildds Vagrant is planning to setup would also useful to test Debian packages (incl uboot) on armhf. cheers, Holger signature.asc Description: This is a digitally signed message part. ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: [Reproducible-builds] [U-Boot] [PATCH] build: create time and date independent binary
Le dimanche 19 juillet 2015 à 06:14 -0700, Vagrant Cascadian a écrit : On 2015-07-19, Holger Levsen wrote: All this said, if you send me patches, I will probably deploy them as I'm very curious and more reproducibility efforts are good :-) We can can always decide to remove or move them later. I wish to make all contributions upstream. What would really help at first would be to have all targets built regularly to see where work is needed. This is where I think the Debian infrastructure could help, in a similar way as what was started for Coreboot. FWIW, I was planning on including this patch to u-boot in the next upload to Debian: https://anonscm.debian.org/cgit/collab-maint/u-boot.git/tree/debian/patches/use-date-from-debian-changelog.patch?h=experimental-2015.07 I *think* that actually makes u-boot build reproducibly with Debian's reproducible builds toolchain when SOURCE_DATE_EPOCH is set, but I haven't tested it fully. I might have missed some other sources of non-determinism... We came up with something similar last week. This is not actually sufficient, but close. I will submit what I have to the U-Boot mailing list soon (hopefully, later today), feel free to review it there and import it to Debian. Hoping to get some armhf buildd nodes up an running soonish... although it should also be buildable with the cross-toolchains, if the reproducible buildds coulld be made to support that. Native builds don't seem realistic given the many different architectures that U-Boot supports, unless done with virtualisation (see my other email from today). Now that I think about it, maybe virtualisation would require simply too many VMs to setup. Either way, we can always try to add a toolchain build script to U-Boot. -- Paul Kocialkowski, Replicant developer Replicant is a fully free Android distribution running on several devices, a free software mobile operating system putting the emphasis on freedom and privacy/security. Website: http://www.replicant.us/ Blog: http://blog.replicant.us/ Wiki/tracker/forums: http://redmine.replicant.us/ signature.asc Description: This is a digitally signed message part ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: [Reproducible-builds] [U-Boot] [PATCH] build: create time and date independent binary
sorry for the late reply. No problem, I haven't kept you or the list posted on this either. I got a chance to discuss the issue with Lunar during RMLL 2015 and we came up with a nice way of doing things, using SOURCE_DATE_EPOCH. On Samstag, 13. Juni 2015, Paul Kocialkowski wrote: you've seen https://reproducible.debian.net/u-boot ? This seems very minimalistic, but it's good to see U-Boot was given some attention already! As far as I understood, this was only for the U-Boot tools. but maybe you can explain why u-boot needs more reproducibility testing than what there currently is. i'm definitly interested and not opposed, even though I think there shoukd be good reasons to treat some software specially. The point is to make U-Boot reproducible for all possible targets, not only the few ones that are supported by U-Boot. I think your sentence is missing some word…?!? not only for the few one that are supported by U-Boot maybe? Looks good otherwise! I think this requires some extra infrastructure. In that sense, it is very similar to Coreboot. (also please note that we currently only have amd64 hw to run our tests on.) The problem is the same as Coreboot, which uses its own toolchain to build images. We don't need to have native armhf builds for U-Boot, testing with the armhf toolchain that is in Debian should be enough. I see. There seem to be two solutions to this: * Including a script (e.g. the one from coreboot) to build the toolchain as part of the build process * Using native builds with visualization on the servers that run the builds for the reproducible task force I tend to prefer the second one since it only requires a one-time setup cost, while the other one, that requires to build toolchains for each test build, implies a considerably longer build time for each test. I understand, this works out nicely because all the work on Coreboot will be inherited by Libreboot. However, on U-Boot, the work to bring reproducible builds has to take place initially. I know for a fact that parts of the code use things like __FILE__ or timestamps. Ah. Not all the targets are using that though, and the target we used during RMLL required only timestamp changes to become reproducible (it was the Cubieboard2 IIRC). Still, I have seen that code around, so it must be used somewhere, so it should be fixed, too. All this said, if you send me patches, I will probably deploy them as I'm very curious and more reproducibility efforts are good :-) We can can always decide to remove or move them later. I wish to make all contributions upstream. What would really help at first would be to have all targets built regularly to see where work is needed. This is where I think the Debian infrastructure could help, in a similar way as what was started for Coreboot. can you point me to a how to explaining this or tell me those steps, starting with git clone...? The basics for building U-Boot are the following (e.g. for the Cubieboard2 target) git clone git://git.denx.de/u-boot.git cd u-boot make -C $SRC O=$DST CROSS_COMPILE=$CROSS_COMPILE Cubieboard2_defconfig make -C $SRC O=$DST CROSS_COMPILE=$CROSS_COMPILE I usually also pass ARCH=$ARCH to make but that's useless, apparently. Let me know if you need more indications on this. -- Paul Kocialkowski, Replicant developer Replicant is a fully free Android distribution running on several devices, a free software mobile operating system putting the emphasis on freedom and privacy/security. Website: http://www.replicant.us/ Blog: http://blog.replicant.us/ Wiki/tracker/forums: http://redmine.replicant.us/ signature.asc Description: This is a digitally signed message part ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
[Reproducible-builds] [U-Boot] [PATCH] build: create time and date independent binary
Thanks for sharing the patches to set U_BOOT_DATE/U_BOOT_TIME/U_BOOT_TZ. I happened to independently work on a similar patch in the last few days: Index: u-boot/Makefile === --- u-boot.orig/Makefile +++ u-boot/Makefile @@ -1231,10 +1231,14 @@ define filechk_version.h echo \#define LD_VERSION_STRING \$$($(LD) --version | head -n 1)\; ) endef +ifeq ($(BUILD_DATE),) + BUILD_DATE = $(shell date) +endif + define filechk_timestamp.h - (LC_ALL=C date +'#define U_BOOT_DATE %b %d %C%y'; \ - LC_ALL=C date +'#define U_BOOT_TIME %T'; \ - LC_ALL=C date +'#define U_BOOT_TZ %z') + (LC_ALL=C date --utc --date $(BUILD_DATE) +'#define U_BOOT_DATE %b %d %C%y'; \ + LC_ALL=C date --utc --date $(BUILD_DATE) +'#define U_BOOT_TIME %T'; \ + LC_ALL=C date --utc --date $(BUILD_DATE) +'#define U_BOOT_TZ %z') endef $(version_h): include/config/uboot.release FORCE This allows us to set a specific timestamp by passing BUILD_DATE (or whatever standardized variable name is agreed upon) in debian/rules taken from the debian/changelog entry. I do see some value in there being a time-based string there, as it's displayed to the user at boot and can be useful for troubleshooting what build they're booting... I'll likely include this patch or soemthing like it in the next u-boot uploads to Debian; though it'd be great to get it merged upstream. Interestingly, u-boot kind of flys under the reproducible builds radar as the amd64 build doesn't include any binaries impacted by the datestrings. I've been making mumblings on irc of setting up some armhf nodes for testing reproducible builds on armhf to catch some of these packages... live well, vagrant signature.asc Description: PGP signature ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
