[Reproducible-builds] debian-installer: builds unreproducible netboot images

2015-11-16 Thread Steven Chamberlain
Package: debian-installer
Version: 20150422
Severity: wishlist
Tags: patch
User: reproducible-builds@lists.alioth.debian.org
Usertag: timestamps fileordering infrastructure
X-Debbugs-Cc: reproducible-builds@lists.alioth.debian.org
X-Debbugs-Cc: debian-...@lists.debian.org

Hi!

The debian-installer package build produces netboot.tar.gz and
the mini.iso netboot install media.  It doesn't do this in an easily
reproducible way:

  * the d-i initrd/mfsroot is a filesystem image, having variable
mtime/ctime/atime timestamps from package build time;
  * likewise in the generated mini.iso;
  * netboot.tar.gz also has varying timestamps;  the order of files
may also vary depending on the filesystem;
  * likewise in the cd info tarball;
  * likewise in the debian-installer-images tarball;
  * all gzipped outfile files have a timestamp in the header.

I have a patch aimed at jessie-kfreebsd that should fix all of the
above.  It should be possible to do the same in sid with much less
code, due to new GNU tar features and other reproducible builds work.

I've 'clamped' timestamps to be no later than the most recent
debian/changelog entry date.  That way, the non-useful timestamps
from during the build are adjusted to a constant value.  Older
timestamps, actually indicating how old a file is, are untouched.
The BUILD_DATE, actually the package version number, is unchanged.

Specifically on kfreebsd, the generated mfsroot is a ffs filesystem
having file atimes, and another timestamp in the filesystem superblock.
I intend to patch makefs so that it can clamp timestamps to a given
SOURCE_DATE_EPOCH.

Besides a file ordering issue in makefs, all output files including
netboot.tar.gz and mini.iso then seem to be reproducible for
jessie-kfreebsd, at least.  :)

Regards,
-- 
Steven Chamberlain
ste...@pyro.eu.org


signature.asc
Description: Digital signature
___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

Re: [Reproducible-builds] debian-installer: builds unreproducible netboot images

2015-11-16 Thread Steven Chamberlain
Attached is my jessie-kfreebsd implementation.  As I said, it should be
much cleaner to implement this in sid with newer GNU tar.

Regards,
-- 
Steven Chamberlain
ste...@pyro.eu.org
diff --git a/build/Makefile b/build/Makefile
index ec5a084..6261a4d 100644
--- a/build/Makefile
+++ b/build/Makefile
@@ -56,7 +56,7 @@
 # Add to PATH so dpkg will always work, and so local programs will be found.
 PATH := util:$(PATH):/usr/sbin:/sbin
 EATMYDATA = $(shell which eatmydata 2>/dev/null)
-GZIP = $(shell which pigz gzip | head -1)
+GZIP = $(shell which pigz gzip | head -1) -n
 
 # We don't want this to be run each time we re-enter.
 ifndef DEB_HOST_ARCH
@@ -149,7 +149,7 @@ MFSROOT_LIMIT := 68m
 endif
 
 define mkfs.ufs1
-  sh -c 'makefs -t ffs -s $(MFSROOT_LIMIT) -f 3000 -o minfree=0,version=1 $$0 ${TREE}'
+  sh -c 'makefs -t ffs -T $(SOURCE_DATE_EPOCH) -s $(MFSROOT_LIMIT) -f 3000 -o minfree=0,version=1 $$0 ${TREE}'
 endef
 
 define e2fsck
@@ -803,7 +803,14 @@ $(TEMP_MINIISO): $(TEMP_BOOT_SCREENS) arch_miniiso
 
 # various kinds of information, for use on debian-cd isos
 $(DEBIAN_CD_INFO): $(TEMP_BOOT_SCREENS) $(TEMP_CD_INFO_DIR)
-	(cd $(TEMP_CD_INFO_DIR); tar czf - .) > $@
+	# Clamp timestamps to be no newer than last changelog entry, see
+	# https://wiki.debian.org/ReproducibleBuilds/TimestampsInTarball
+	find $(TEMP_CD_INFO_DIR) -newermt "@$(SOURCE_DATE_EPOCH)" -print0 | xargs -0r touch --no-dereference --date="@$(SOURCE_DATE_EPOCH)"
+	# Create tarball with files sorted in a stable order, see
+	# https://wiki.debian.org/ReproducibleBuilds/FileOrderInTarballs
+	# and without timestamp in the gzip header, see
+	# https://wiki.debian.org/ReproducibleBuilds/TimestampsInGzipHeaders
+	( cd $(TEMP_CD_INFO_DIR) && find . -print0 | LC_ALL=C sort -z | GZIP=-n tar --no-recursion --null -T - -czf -) > $@
 	update-manifest $@ $(MANIFEST-DEBIAN_CD_INFO)
 
 # a directory full of files for netbooting
@@ -822,7 +829,14 @@ $(NETBOOT_TAR): $(TEMP_NETBOOT_DIR)
 	# Create an version info file.
 	echo 'Debian version:  $(DEBIAN_VERSION)' > $(TEMP_NETBOOT_DIR)/version.info
 	echo 'Installer build: $(BUILD_DATE)' >> $(TEMP_NETBOOT_DIR)/version.info
-	(cd $(TEMP_NETBOOT_DIR); tar czf - .) > $@
+	# Clamp timestamps to be no newer than last changelog entry, see
+	# https://wiki.debian.org/ReproducibleBuilds/TimestampsInTarball
+	find $(TEMP_NETBOOT_DIR) -newermt "@$(SOURCE_DATE_EPOCH)" -print0 | xargs -0r touch --no-dereference --date="@$(SOURCE_DATE_EPOCH)"
+	# Create tarball with files sorted in a stable order, see
+	# https://wiki.debian.org/ReproducibleBuilds/FileOrderInTarballs
+	# and without timestamp in the gzip header, see
+	# https://wiki.debian.org/ReproducibleBuilds/TimestampsInGzipHeaders
+	( cd $(TEMP_NETBOOT_DIR) && find . -print0 | LC_ALL=C sort -z | GZIP=-n tar --no-recursion --null -T - -czf -) > $@
 	update-manifest $@ $(MANIFEST-NETBOOT_TAR) $(UDEB_LISTS)
 
 $(TEMP_BOOT_SCREENS): arch_boot_screens
diff --git a/build/config/x86.cfg b/build/config/x86.cfg
index 3caadd2..b0fc9a2 100644
--- a/build/config/x86.cfg
+++ b/build/config/x86.cfg
@@ -332,6 +332,11 @@ arch_miniiso: x86_syslinux x86_grub_efi
 			| todos > $(TEMP_CD_TREE)/win32-loader.ini; \
 	fi
 
+	# Clamp timestamps to be no newer than last changelog entry, see
+	# https://wiki.debian.org/ReproducibleBuilds/TimestampsInTarball
+	find $(TEMP_CD_TREE) -newermt "$(SOURCE_DATE)" -print0 \
+	 | xargs -0r touch --no-dereference --date="$(SOURCE_DATE)"
+
 	if [ "$(GRUB_EFI)" = y ]; then \
 		xorriso -as mkisofs -r -J -b isolinux.bin -c boot.cat \
 			-no-emul-boot -boot-load-size 4 -boot-info-table \
diff --git a/debian/changelog b/debian/changelog
index 42aed37..09c8a02 100644
--- a/debian/changelog
+++ b/debian/changelog
@@ -1,3 +1,21 @@
+debian-installer (20150422+kbsd8u2) jessie-kfreebsd; urgency=medium
+
+  * Improve reproducibility of debian-installer netboot images:
+(Closes: #805321)
+- clamp timestamps in the d-i ramdisk to be no later than
+  the most recent debian/changelog entry of this package
+  - raise makefs dependency on >= 20100306-5+kbsd8u1
+- clamp timestamps in the mini.iso similarly
+- clamp timestamps in the netboot tarball;  store files in a
+  stable order
+- clamp timestamps in the cd info tarball;  store files in a
+  stable order
+- clamp timestamps in the output debian-installer-images tarball;
+  store files in a stable order
+- disable timestamps in gzip output (e.g. initrd.gz and tarballs)
+
+ -- Steven Chamberlain   Tue, 10 Nov 2015 21:38:46 +
+
 debian-installer (20150422+kbsd8u1) jessie-kfreebsd; urgency=medium
 
   * Rebuild using udebs from the jessie-kfreebsd suite, also using
diff --git a/debian/control b/debian/control
index 100ca5a..6f4df5b 100644
--- a/debian/control
+++ b/debian/control
@@ -162,7 +162,7 @@ Build-Depends:
 #		architectures if SSL_CERTS has been set locally.
 	win32-loader (>= 0.7.2) [i386 amd64 kfreebsd-i386 kfreebsd-amd64 hurd-i386],
 #		Alternative boot method