Bastian Blank:
> On Mon, Jan 05, 2015 at 06:56:10PM +0100, Jérémy Bobbio wrote:
> > > > >  * linux-source: mtimes of many files differ. Would it be ok to just
> > > > >    create the tarball with a single timestamp (`tar --mtime=`)?
> > I've used find+touch instead. See attached patch.
> 
> Why?

Good question. It's the way we are doing it in dh_builddeb. But for
kernel-source, using `--mtime` is more straight forward. This is done in
the attached patch.

> > I also stumbled on a variation of the Installed-Size field in one of the
> > .deb. But this is a different topic.
> 
> Is this value computed in a stable way?

The value can differ greatly depending on the underlying filesystem.
This is to be addressed at dpkg's level. josch is working on it.

> > +@@ -301,7 +301,9 @@ if [ ! -z ${output_file} ]; then
> > +   if [ -z ${cpio_file} ]; then
> > +           timestamp=
> > +           if test -n "$KBUILD_BUILD_TIMESTAMP"; then
> > +-                  timestamp="$(date -d"$KBUILD_BUILD_TIMESTAMP" +%s || :)"
> > ++                  source_date=$(echo "$KBUILD_BUILD_TIMESTAMP" |
> > ++                                  sed -e 's/.*(\([0-9-]\+\)).*/\1/')
> > ++                  timestamp="$(date -d"$source_date" +%s || :)"
> 
> This solution may not work.  The patched source can be built with a
> normal timestamp override, via linux-source.

The above construction will work given a standard date in
KBUILD_BUILD_TIMESTAMP. The sed expression only match parenthesises.
I am open to other suggestions.

> > @@ -87,6 +87,8 @@
> >     rm -rf '$@' '$(DIR)'
> >     $(call copy_source,$(DIR))
> >     chmod -R u+rw,go=rX '$(DIR)'
> > +   find '$(DIR)' -depth -newermt '$(SOURCE_DATE)' -print0 | \
> > +           xargs -0r touch --no-dereference --date='$(SOURCE_DATE)'
> 
> -newermt?  This does not look really stable.

I don't understand what you mean by that. It's been proven working for
a while in our experiments.


With the attached patch, linux can be built reproducibly with the
current experimental framework. The following changes have been
introduced since the previous patch:

 * Files in the source tarball are added in a stable order.
 * kernel-doc is made deterministic when parsing `@foo()` in
   descriptions.
 * The last set of manpages when generating manpages from files
   referenced in multiple sections of the documentation is
   deterministic.

Some `@foo()` constructs in the kernel should probably be fixed to be
just `foo()`. It's unclear what should be used for functions passed as
parameters.

-- 
Lunar                                .''`. 
lu...@debian.org                    : :Ⓐ  :  # apt-get install anarchism
                                    `. `'` 
                                      `-   
diff -Nru linux-3.16.7-ckt2/debian/changelog linux-3.16.7-ckt2/debian/changelog
--- linux-3.16.7-ckt2/debian/changelog	2014-12-08 21:03:19.000000000 +0100
+++ linux-3.16.7-ckt2/debian/changelog	2015-01-07 16:50:40.000000000 +0100
@@ -1,3 +1,16 @@
+linux (3.16.7-ckt2-1.0~reproducible1) UNRELEASED; urgency=low
+
+  * Make build reproducible:
+    - Add a patch to make gen_initramfs.sh parse Debian's KBUILD_BUILD_TIMESTAMP.
+    - Add a patch to make kernel-doc parse highlights determistically.
+    - Add a patch to force the order when generating manpages from files
+      referenced in multiple sections of the documentation.
+    - Use debian/changelog date as mtimes in the generated source tarball.
+    - Write files in the source tarball in a stable order.
+    - Stop recording current time in gzip headers when compressing documentation.
+
+ -- Jérémy Bobbio <lu...@debian.org>  Wed, 07 Jan 2015 16:50:39 +0100
+
 linux (3.16.7-ckt2-1) unstable; urgency=high
 
   * New upstream stable update:
diff -Nru linux-3.16.7-ckt2/debian/patches/debian/generate-manpage-in-deterministic-order.patch linux-3.16.7-ckt2/debian/patches/debian/generate-manpage-in-deterministic-order.patch
--- linux-3.16.7-ckt2/debian/patches/debian/generate-manpage-in-deterministic-order.patch	1970-01-01 01:00:00.000000000 +0100
+++ linux-3.16.7-ckt2/debian/patches/debian/generate-manpage-in-deterministic-order.patch	2015-01-07 16:34:00.000000000 +0100
@@ -0,0 +1,55 @@
+Description: generate manpages in deterministic order
+ When header files are referenced multiple times in different
+ documentation parts, manpages for the various functions will also
+ be generated multiple times.
+ .
+ In order to support reproducible builds when multiple jobs are
+ enabled, we need to make the order deterministic. To do so, we
+ scan the documentation, identify sections having similar directives,
+ and generate dependencies making the first section in alphabetical
+ order build last.
+ .
+ With the current kernel, the following dependencies are generated:
+ .
+     Documentation/DocBook/alsa-driver-api.tmpl: Documentation/DocBook/device-drivers.tmpl
+     Documentation/DocBook/gadget.tmpl: Documentation/DocBook/usb.tmpl
+     Documentation/DocBook/genericirq.tmpl: Documentation/DocBook/kernel-api.tmpl
+Author: Jérémy Bobbio <lu...@debian.org>
+
+--- linux-3.16.7-ckt2.orig/Documentation/DocBook/Makefile
++++ linux-3.16.7-ckt2/Documentation/DocBook/Makefile
+@@ -92,6 +92,24 @@ endef
+ %.xml: %.tmpl $(KERNELDOC) $(DOCPROC) FORCE
+ 	$(call if_changed_rule,docproc)
+ 
++# Ensure a deterministic build order when the same file are sourced in
++# different locations
++TEMPLATES = $(patsubst %.xml,%.tmpl,$(BOOKS))
++$(obj)/.docbook.d: $(TEMPLATES)
++	@set -e;							\
++	cd $(srctree);							\
++	grep '^!' $(TEMPLATES) |					\
++		tr ':' '\t' | LC_ALL=C sort -k2 |			\
++		awk '   { if (previous_include == $$2 && previous_source != $$1) { \
++				print previous_source ": " $$1;		\
++			} else {					\
++				previous_source = $$1;			\
++				previous_include = $$2;			\
++			} }' | sort -u |				\
++		sed -e 's/\.tmpl/\.9/g' > $@
++
++-include $(obj)/.docbook.d
++
+ # Tell kbuild to always build the programs
+ always := $(hostprogs-y)
+ 
+@@ -218,7 +236,8 @@ clean-files := $(DOCBOOKS) \
+ 	$(patsubst %.xml, %.pdf,  $(DOCBOOKS)) \
+ 	$(patsubst %.xml, %.html, $(DOCBOOKS)) \
+ 	$(patsubst %.xml, %.9,    $(DOCBOOKS)) \
+-	$(index)
++	$(index) \
++	$(obj)/.docbook.d
+ 
+ clean-dirs := $(patsubst %.xml,%,$(DOCBOOKS)) man
+ 
diff -Nru linux-3.16.7-ckt2/debian/patches/debian/make-highlights-deterministic-in-kernel-doc.patch linux-3.16.7-ckt2/debian/patches/debian/make-highlights-deterministic-in-kernel-doc.patch
--- linux-3.16.7-ckt2/debian/patches/debian/make-highlights-deterministic-in-kernel-doc.patch	1970-01-01 01:00:00.000000000 +0100
+++ linux-3.16.7-ckt2/debian/patches/debian/make-highlights-deterministic-in-kernel-doc.patch	2015-01-07 16:43:31.000000000 +0100
@@ -0,0 +1,21 @@
+Description: parse kernel-doc deterministically
+ Regular expressions for highlights in kernel-doc are stored in a Perl
+ hash. These hashes are ordered differently for each Perl run. This will
+ prevent kernel-doc to behave deterministically when parsing
+ “@foo()” as in some runs it will be interpreted as a parameter and
+ in the others it will be interpreted as a function.
+ .
+ We now sort the %highlights hash to get the same behavior on every run.
+Author: Jérémy Bobbio <lu...@debian.org>
+
+--- linux-3.16.7-ckt2.orig/scripts/kernel-doc
++++ linux-3.16.7-ckt2/scripts/kernel-doc
+@@ -2586,7 +2586,7 @@ $kernelversion = get_kernel_version();
+ 
+ # generate a sequence of code that will splice in highlighting information
+ # using the s// operator.
+-foreach my $pattern (keys %highlights) {
++foreach my $pattern (sort keys %highlights) {
+ #   print STDERR "scanning pattern:$pattern, highlight:($highlights{$pattern})\n";
+     $dohighlight .=  "\$contents =~ s:$pattern:$highlights{$pattern}:gs;\n";
+ }
diff -Nru linux-3.16.7-ckt2/debian/patches/debian/parse-debian-build-timestamp.patch linux-3.16.7-ckt2/debian/patches/debian/parse-debian-build-timestamp.patch
--- linux-3.16.7-ckt2/debian/patches/debian/parse-debian-build-timestamp.patch	1970-01-01 01:00:00.000000000 +0100
+++ linux-3.16.7-ckt2/debian/patches/debian/parse-debian-build-timestamp.patch	2015-01-05 14:50:58.000000000 +0100
@@ -0,0 +1,26 @@
+Description: parse Debian KBUILD_BUILD_TIMESTAMP in gen_initramfs_list.sh
+ gen_initramfs_list.sh assumes that KBUILD_BUILD_TIMESTAMP only contains a
+ date. In Debian, it's a longer string looking like:
+ “Debian 3.16.7-2 (2014-11-06)”. Here we adapt the script to parse Debian
+ format.
+ .
+ This change is necessary to have reproducible builds as the date will
+ then be used as mtime for files in initramfs archive instead of the
+ current time.
+Author: Jérémy Bobbio <lu...@debian.org>
+Bug-Debian: https://bugs.debian.org/769844
+Forwarded: not-needed
+
+--- linux-3.16.7-ckt2.orig/scripts/gen_initramfs_list.sh
++++ linux-3.16.7-ckt2/scripts/gen_initramfs_list.sh
+@@ -301,7 +301,9 @@ if [ ! -z ${output_file} ]; then
+ 	if [ -z ${cpio_file} ]; then
+ 		timestamp=
+ 		if test -n "$KBUILD_BUILD_TIMESTAMP"; then
+-			timestamp="$(date -d"$KBUILD_BUILD_TIMESTAMP" +%s || :)"
++			source_date=$(echo "$KBUILD_BUILD_TIMESTAMP" |
++					sed -e 's/.*(\([0-9-]\+\)).*/\1/')
++			timestamp="$(date -d"$source_date" +%s || :)"
+ 			if test -n "$timestamp"; then
+ 				timestamp="-t $timestamp"
+ 			fi
diff -Nru linux-3.16.7-ckt2/debian/patches/series linux-3.16.7-ckt2/debian/patches/series
--- linux-3.16.7-ckt2/debian/patches/series	2014-12-08 21:00:20.000000000 +0100
+++ linux-3.16.7-ckt2/debian/patches/series	2015-01-07 16:34:21.000000000 +0100
@@ -7,6 +7,9 @@
 debian/arch-sh4-fix-uimage-build.patch
 debian/powerpcspe-omit-uimage.patch
 features/all/Kbuild-kconfig-Verbose-version-of-listnewconfig.patch
+debian/parse-debian-build-timestamp.patch
+debian/make-highlights-deterministic-in-kernel-doc.patch
+debian/generate-manpage-in-deterministic-order.patch
 
 # Fixes/improvements to firmware loading
 features/all/drivers-media-dvb-usb-af9005-request_firmware.patch
--- linux-3.16.7-ckt2/debian/rules.real	2014-11-04 05:41:34.000000000 +0100
+++ linux-3.16.7-ckt2/debian/rules.real	2015-01-07 14:37:52.000000000 +0100
@@ -87,7 +87,9 @@
 	rm -rf '$@' '$(DIR)'
 	$(call copy_source,$(DIR))
 	chmod -R u+rw,go=rX '$(DIR)'
-	cd '$(BUILD_DIR)'; tar -caf 'linux-source-$(UPSTREAMVERSION).tar.xz' 'linux-source-$(UPSTREAMVERSION)'
+	cd '$(BUILD_DIR)'; find 'linux-source-$(UPSTREAMVERSION)' -type f -print0 | \
+		LC_ALL=C sort -z | tar --mtime='$(SOURCE_DATE)' --null -T - \
+			-caf 'linux-source-$(UPSTREAMVERSION).tar.xz' 
 	rm -rf '$(DIR)'
 
 $(BUILD_DIR)/linux-patch-$(UPSTREAMVERSION)-%.patch.xz: $(STAMPS_DIR)/source_none $(STAMPS_DIR)/source_%
@@ -202,7 +204,7 @@
 	cd $(DIR)/Documentation/DocBook; \
 	find * -name '*.html' -print | \
 	cpio -pd --preserve-modification-time '$(CURDIR)/$(OUT_DIR)/html'
-	gzip -9qfr $(OUT_DIR)/Documentation
+	gzip -9nqfr $(OUT_DIR)/Documentation
 	+$(MAKE_SELF) install-base
 
 install-manual: PACKAGE_NAME = linux-manual-$(VERSION)

Attachment: signature.asc
Description: Digital signature

_______________________________________________
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

Reply via email to