Hi there! This is a long rambling email, sorry.

I ran across the OpenWRT reproducible build page[0] and diffoscope outputs this morning and thought they were super rad. I spent a bit of time to see if I could push things forward. Presumably this will mostly involve interaction with the OpenWRT developers, but I wanted to check in here first for feedback and to share what I learned.

1. There's a statement that diffoscope "cannot detect .bin files as squashfs". I don't think that's accurate, I could rename .squashfs files to .bin and diffoscope found the magic and worked fine. I think the issue is the huge number of hardware-specific ".bin" and ".img" formats OpenWRT generates[1]. Many of these are in uImage format (detectable by magic), others are just {kernel, bootloader, fs} at fixed offsets. For the former it would be great if diffoscope could pick up on the format in any way. I found a random script[2] that tries to extract uImage binaries into the original files, but it would be a good start to just present the metadata and offsets.

2. A few small changes to reproducible_openwrt.sh would be helpful. I would enable .tar.gz rootfs output for all targets (CONFIG_TARGET_ROOTFS_TARGZ=y) and grab some extra "image" files. In addition to .bin and .squashfs, {*.img, *.ext4, *.rootfs.tar.gz, *.img.gz, *.ubi, *.ubifs, *.itb} are full image suffixes. {*-uImage, *-zImage, *-vmlinuz, *.dtb, *.elf} are suffixes of kernel files which are sometimes distributed separately.
Also, perhaps add CONFIG_CLEAN_IPKG=y (see below).
Also, perhaps an ARM build target; A20-OLinuXino-Lime? Something with device tree, otherwise doesn't really matter. I can submit a patch for the above if that would help, but I wouldn't have an easy way to test it.

In particular, being able to compare the rootfs images and kernel images separately might be sufficient instead of trying to untangle all the many random combined binary image formats.

3. I was sort of surprised to see linux-latest passing reproducibility in Debian on armhf. Was anything special necessary for this, eg changes to mkimage to generate uImage without a timestamp? Any advice or HOWTO out there on reproducible kernel builds? I see Ben Hutchings's kbuild patch for SOURCE_DATE_EPOCH support; is that sufficient?

4. busybox uses kconfig (or something very similar), which generates a AUTOCONF_TIMESTAMP macro in a header file (passed through as a "banner" variable capturing build time). I assumed this had to do with autotools at first, but it seems to be an old kconfig thing. I don't know how many other projects use these tools, but it might be worth making an identifier. I worked around the problem by checking for SOURCE_DATE_EPOCH in the Makefile and overriding the macro, but it might be cleaner to check in kconfig itself and generate an appropriate macro variable instead, as a reusable pattern.

5. diffoscope has squashfs support, but is there actually a way to generate squashfs files reproducibly? Perhaps there is a way to strip the metadata after the fact?

6. The opkg package manger leaves a log of .ipkg installs which end up on the disk and include a (UNIX) Installed-Time field. This situation is very similar to /var/cache with Debian. The CONFIG_CLEAN_IPKG=y flag is sufficient to just delete this log file, or opkg could add an argument to override the time, or a sed script could "fix" all the timestamps. A question for the OpenWRT developers.

I haven't gotten to any complete reproducible image files yet; I've got some patches with partial progress here: https://github.com/bnewbold/openwrt-repro/compare/master...repro

These are mostly just passing an existing TIMESTAMP variable (from version control) through to tar, having gzip not make timestamps, sorting a text file, etc.

The next things to tackle would be kernel images and a few more packages that capture timestamps. Also, with squashfs (vs tar), device files like /dev/console are problematic because they can't be "touched" to change mtime for mksquashfs.

Thanks for not only pushing through so much progress within Debian but also setting up this example infrastructure for OpenWRT, BSDs, etc. Inspiring!


[0]: https://reproducible.debian.net/openwrt/openwrt.html
[1]: http://wiki.openwrt.org/doc/howto/obtain.firmware.generate
[2]: http://buffalo.nas-central.org/wiki/How_to_Extract_an_uImage

Reproducible-builds mailing list

Reply via email to