Bug#806945: bash: Please make bash build reproducibly
Ximin Luo: > [..] > > CFLAGS - we eventually want to patch gcc to fix this; bash doesn't need to > handle this specifically. > BUILD_DIR - this does seem bash-specific, but I haven't yet studied the > Makefiles in enough detail to propose something concrete. > PDF ID - this is caused by [1] and we will probably patch texlive-bin to fix > this; bash doesn't need to handle this specifically. > Hi, here is an updated patch that fixes the BUILD_DIR issue. My patch to GCC (#862113) fixes the CFLAGS issue here, as expected. The only remaining issue is the PDF ID issue, which I'll work on in the near-to-mid future. i.e. this patch is (hopefully) the complete set of changes that need to go into the bash package, for reproducibility purposes. Together with the GCC patch and a future texlive package, these will make the bash package fully reproducible under all build environments. X -- GPG: ed25519/56034877E1F87C35 GPG: rsa4096/1318EFAC5FBBDBCE https://github.com/infinity0/pubkeys.git diff -Nru bash-4.4/debian/changelog bash-4.4/debian/changelog --- bash-4.4/debian/changelog 2017-01-24 16:13:37.0 +0100 +++ bash-4.4/debian/changelog 2016-09-26 14:40:45.0 +0200 @@ -1,3 +1,17 @@ +bash (4.4-4.0~reproducible1) UNRELEASED; urgency=medium + + [ Ximin Luo ] + * Non-maintainer upload. + * Set PGRP_PIPE unconditionally on Linux, reproducibly. + * Don't embed @BUILD_DIR@ in loadables example Makefile, so we can build +reproducibly. Instead, pass in the real value from the parent Makefile. + + [ Reiner Herrmann ] + * Use the system man2html instead of the embedded one, for better build +reproducibility. + + -- Ximin LuoMon, 26 Sep 2016 14:40:45 +0200 + bash (4.4-4) unstable; urgency=medium * Apply upstream patches 008 - 011. diff -Nru bash-4.4/debian/control bash-4.4/debian/control --- bash-4.4/debian/control 2017-01-20 11:52:59.0 +0100 +++ bash-4.4/debian/control 2016-09-26 14:40:45.0 +0200 @@ -6,7 +6,7 @@ Build-Depends: autoconf, autotools-dev, bison, libncurses5-dev, texinfo, texi2html, debhelper (>= 9), gettext, sharutils, locales , time , - xz-utils, dpkg-dev (>= 1.16.1) + xz-utils, dpkg-dev (>= 1.16.1), man2html Build-Depends-Indep: texlive-latex-base, ghostscript, texlive-fonts-recommended Homepage: http://tiswww.case.edu/php/chet/bash/bashtop.html Vcs-Browser: https://code.launchpad.net/~doko/+junk/pkg-bash-debian diff -Nru bash-4.4/debian/patches/no-buildpath-in-runtime-example-makefile.diff bash-4.4/debian/patches/no-buildpath-in-runtime-example-makefile.diff --- bash-4.4/debian/patches/no-buildpath-in-runtime-example-makefile.diff 1970-01-01 01:00:00.0 +0100 +++ bash-4.4/debian/patches/no-buildpath-in-runtime-example-makefile.diff 2016-09-26 14:40:45.0 +0200 @@ -0,0 +1,101 @@ +Index: bash-4.4/examples/loadables/Makefile.in +=== +--- bash-4.4.orig/examples/loadables/Makefile.in bash-4.4/examples/loadables/Makefile.in +@@ -35,7 +35,7 @@ loadablesdir = @loadablesdir@ + headersdir = @headersdir@ + + topdir = @top_srcdir@ +-BUILD_DIR = @BUILD_DIR@ ++BUILD_DIR = $(CURDIR) + srcdir = @srcdir@ + VPATH = @srcdir@ + +Index: bash-4.4/examples/loadables/Makefile.inc.in +=== +--- bash-4.4.orig/examples/loadables/Makefile.inc.in bash-4.4/examples/loadables/Makefile.inc.in +@@ -38,7 +38,7 @@ loadablesdir = @loadablesdir@ + headersdir = @headersdir@ + + topdir = @top_srcdir@ +-BUILD_DIR = @BUILD_DIR@ ++BUILD_DIR = $(CURDIR) + srcdir = @srcdir@ + VPATH = @srcdir@ + +Index: bash-4.4/Makefile.in +=== +--- bash-4.4.orig/Makefile.in bash-4.4/Makefile.in +@@ -561,6 +561,7 @@ OTHER_DOCS = $(srcdir)/CHANGES $(srcdir) + OTHER_INSTALLED_DOCS = CHANGES COMPAT NEWS POSIX RBASH README + + LOADABLES_DIR = ${top_builddir}/examples/loadables ++LOADABLES_MAKE = cd $(LOADABLES_DIR) && $(MAKE) BUILD_DIR=$(BUILD_DIR) + + # Keep GNU Make from exporting the entire environment for small machines. + .NOEXPORT: +@@ -757,7 +758,7 @@ reconfig: force + sh $(srcdir)/configure -C + + loadables: +- cd $(LOADABLES_DIR) && $(MAKE) $(MFLAGS) all ++ $(LOADABLES_MAKE) $(MFLAGS) all + + #newversion: mkversion + # $(RM) .build +@@ -799,7 +800,7 @@ install: .made installdirs + infodir=$(infodir) htmldir=$(htmldir) DESTDIR=$(DESTDIR) $@ ) + -( cd $(DEFDIR) ; $(MAKE) $(MFLAGS) DESTDIR=$(DESTDIR) $@ ) + -( cd $(PO_DIR) ; $(MAKE) $(MFLAGS) DESTDIR=$(DESTDIR) $@ ) +- -( cd $(LOADABLES_DIR) && $(MAKE) $(MFLAGS) DESTDIR=$(DESTDIR) $@ ) ++ -( $(LOADABLES_MAKE) $(MFLAGS) DESTDIR=$(DESTDIR) $@ ) + + install-strip: + $(MAKE) $(MFLAGS) INSTALL_PROGRAM='$(INSTALL_PROGRAM) -s' \ +@@ -842,7 +843,7 @@ uninstall: .made + man3dir=$(man3dir) man3ext=$(man3ext) \ + infodir=$(infodir) htmldir=$(htmldir) DESTDIR=$(DESTDIR) $@ ) + -( cd $(PO_DIR) ;
Bug#806945: bash: Please make bash build reproducibly
Package: src:bash Followup-For: Bug #806945 Dear Maintainer, Attached is the patch refreshed for bash version 4.4-1. I have confirmed that it works the same way as advertised previously. As an update, pgrp-pipe.diff is now optional; you can also just drop it to get the same effect, since upstream have patched configure{,.ac} to correctly handle Linux 4. However, I recommend keeping our patch as-is, in case upstream forget to do this again for Linux 5. Outside of this patch, we have also started to vary the build-path during our tests. This has revealed that bash also embeds the build-path into the output; see the attached diffoscope output for details. We don't have a patch for this yet, but in case anyone wants to help here is how we *would* fix it: CFLAGS - we eventually want to patch gcc to fix this; bash doesn't need to handle this specifically. BUILD_DIR - this does seem bash-specific, but I haven't yet studied the Makefiles in enough detail to propose something concrete. PDF ID - this is caused by [1] and we will probably patch texlive-bin to fix this; bash doesn't need to handle this specifically. The other differences are due to the three mentioned above, and will disappear once these are fixed. It would still be good if you applied the attached patch, just to "get it in there" so we don't have to keep refreshing it against newer bash versions. Thanks! Ximin [1] https://sources.debian.net/src/texlive-bin/2016.20160513.41080-6/texk/web2c/pdftexdir/utils.c/#L731 -- System Information: Debian Release: stretch/sid APT prefers testing APT policy: (990, 'testing'), (500, 'stable'), (300, 'unstable'), (200, 'experimental') Architecture: amd64 (x86_64) Foreign Architectures: i386 Kernel: Linux 4.6.0-1-amd64 (SMP w/4 CPU cores) Locale: LANG=en_GB.utf8, LC_CTYPE=en_GB.utf8 (charmap=UTF-8) Shell: /bin/sh linked to /bin/dash Init: systemd (via /run/systemd/system) --- /tmp/tmp16zayndw/control_artifact/ +++ /tmp/tmp16zayndw/experiment_artifact/ ├── bash-builtins_4.4-1.0~reproducible1_amd64.deb │ ├── file list │ │ @@ -1,3 +1,3 @@ │ │ -rw-r--r-- 0004 2016-09-26 12:50:21.00 debian-binary │ │ --rw-r--r-- 000 3680 2016-09-26 12:50:21.00 control.tar.gz │ │ --rw-r--r-- 000 100872 2016-09-26 12:50:21.00 data.tar.xz │ │ +-rw-r--r-- 000 3678 2016-09-26 12:50:21.00 control.tar.gz │ │ +-rw-r--r-- 000 100848 2016-09-26 12:50:21.00 data.tar.xz │ ├── control.tar.gz │ │ ├── control.tar │ │ │ ├── ./md5sums │ │ │ │ ├── md5sums │ │ │ │ │┄ Files in package differ │ │ │ │ ╵ │ │ │ ╵ │ │ ╵ │ ├── data.tar.xz │ │ ├── data.tar │ │ │ ├── file list │ │ │ │ @@ -59,15 +59,15 @@ │ │ │ │ -rw-r--r-- 0 root (0) root (0) 3544 2016-09-26 12:50:21.00 ./usr/include/bash/syntax.h │ │ │ │ -rw-r--r-- 0 root (0) root (0) 2003 2016-09-26 12:50:21.00 ./usr/include/bash/unwind_prot.h │ │ │ │ -rw-r--r-- 0 root (0) root (0)16993 2016-09-26 12:50:21.00 ./usr/include/bash/variables.h │ │ │ │ -rw-r--r-- 0 root (0) root (0) 578 2016-09-26 12:50:21.00 ./usr/include/bash/version.h │ │ │ │ -rw-r--r-- 0 root (0) root (0) 1759 2016-09-26 12:50:21.00 ./usr/include/bash/xmalloc.h │ │ │ │ drwxr-xr-x 0 root (0) root (0)0 2016-09-26 12:50:21.00 ./usr/lib/ │ │ │ │ drwxr-xr-x 0 root (0) root (0)0 2016-09-26 12:50:21.00 ./usr/lib/bash/ │ │ │ │ --rw-r--r-- 0 root (0) root (0) 2875 2016-09-26 12:50:21.00 ./usr/lib/bash/Makefile.inc │ │ │ │ +-rw-r--r-- 0 root (0) root (0) 2881 2016-09-26 12:50:21.00 ./usr/lib/bash/Makefile.inc │ │ │ │ -rwxr-xr-x 0 root (0) root (0) 6352 2016-09-26 12:50:21.00 ./usr/lib/bash/basename │ │ │ │ -rwxr-xr-x 0 root (0) root (0) 6312 2016-09-26 12:50:21.00 ./usr/lib/bash/dirname │ │ │ │ -rwxr-xr-x 0 root (0) root (0)14848 2016-09-26 12:50:21.00 ./usr/lib/bash/finfo │ │ │ │ -rwxr-xr-x 0 root (0) root (0)10512 2016-09-26 12:50:21.00 ./usr/lib/bash/head │ │ │ │ -rwxr-xr-x 0 root (0) root (0)10440 2016-09-26 12:50:21.00 ./usr/lib/bash/id │ │ │ │ -rwxr-xr-x 0 root (0) root (0)10528 2016-09-26 12:50:21.00 ./usr/lib/bash/ln │ │ │ │ -rwxr-xr-x 0 root (0) root (0) 6352 2016-09-26 12:50:21.00 ./usr/lib/bash/logname │ │ │ │ @@ -92,15 +92,15 @@ │ │ │ │ drwxr-xr-x 0 root (0) root (0)0 2016-09-26 12:50:21.00 ./usr/lib/pkgconfig/ │ │ │ │
Bug#806945: bash: Please make bash build reproducibly
Control: tags -1 + patch I've attached the full Debian patch to make bash 4.3-14 reproducible, that includes Reiner's from above. It also includes *fixing a bug in upstream bash*, which currently does-not-appear-in-the-wild *only because* distros already happen to be working around it. But upstream really should fix it - i.e. to fix the PGRP_PIPE check in configure/configure.ac to account for Linux 4+. I decided to just set PGRP_PIPE unconditionally in configure.ac and configure, because I figure nobody will ever again use linux 0, 1 or 2 so it's not worth the extra complexity (which only gives a minor performance gain anyway, as opposed to having *incorrect behaviour*). Upstream is free to choose whichever behaviour he wants - either this simpler version, or the more complex version from my previous email quoted below. Both are correct, if I understand right, and when either is applied to upstream, all distros (including Debian) can drop our specific patches for PGRP_PIPE. To re-iterate again, this does not solve the longer-term issue of "installing config.h is bad". Ximin Ximin Luo: > (Chet, your specific attention is required for this email, please) > > Ximin Luo: >> On Sat, 28 May 2016 13:38:35 +0200 Reiner Herrmann>> wrote: >>> After that, the only remaining issue is that the included header file >>> /usr/include/bash/config.h varies depending on the kernel version used >>> during build [1] (with kernel <4, PGRP_PIPE is defined). >> >> For this particular example, we can just patch this out, i.e. remove it from >> the installed config.h. Debian already forces PGRP_PIPE 1 in config-bot.h, >> which config.h includes at the end. >> > > I dug into this a bit more and it looks like the cause of the difference is > this snippet from configure.ac: > > linux*) LOCAL_LDFLAGS=-rdynamic # allow dynamic loading > case "`uname -r`" in > 2.[[456789]]*|3*) AC_DEFINE(PGRP_PIPE) ;; > esac ;; > > > This was added between bash-3.0.16 and bash-3.1, way before Linux 4 came out. > So I wonder if this snippet should instead be: > > linux*) LOCAL_LDFLAGS=-rdynamic # allow dynamic loading > case "`uname -r`" in > 1.*|2.[[0123]]*) true ;; > *) AC_DEFINE(PGRP_PIPE) ;; > esac ;; > > to set this for all future kernels? Then Debian (and probably other distros) > could get rid of our patch, too. > > However, the question still remains why config.h is installed into the > end-user system, and if bash-built-with-linux-5 required PGRP_PIPE to be > *undefined*, we would still have a reproducibility problem. > > Ximin > -- GPG: ed25519/56034877E1F87C35 GPG: rsa4096/1318EFAC5FBBDBCE git://github.com/infinity0/pubkeys.git diff -Nru bash-4.3/debian/changelog bash-4.3/debian/changelog --- bash-4.3/debian/changelog 2015-09-01 01:04:48.0 +0200 +++ bash-4.3/debian/changelog 2016-06-07 11:56:09.0 +0200 @@ -1,3 +1,15 @@ +bash (4.3-14.0~reproducible1) UNRELEASED; urgency=medium + + [ Ximin Luo ] + * Non-maintainer upload. + * Set PGRP_PIPE unconditionally on Linux, reproducibly. + + [ Reiner Herrmann ] + * Use the system man2html instead of the embedded one, for better build +reproducibility. + + -- Ximin Luo Tue, 07 Jun 2016 11:56:07 +0200 + bash (4.3-14) unstable; urgency=medium * Apply upstream patches 040 - 042. diff -Nru bash-4.3/debian/control bash-4.3/debian/control --- bash-4.3/debian/control 2015-01-28 17:13:32.0 +0100 +++ bash-4.3/debian/control 2016-06-06 03:00:38.0 +0200 @@ -5,7 +5,7 @@ Standards-Version: 3.9.6 Build-Depends: autoconf, autotools-dev, bison, libncurses5-dev, texinfo, texi2html, debhelper (>= 5), locales, gettext, sharutils, time, - xz-utils, dpkg-dev (>= 1.16.1) + xz-utils, dpkg-dev (>= 1.16.1), man2html Build-Depends-Indep: texlive-latex-base, ghostscript, texlive-fonts-recommended Homepage: http://tiswww.case.edu/php/chet/bash/bashtop.html Vcs-Browser: https://code.launchpad.net/~doko/+junk/pkg-bash-debian diff -Nru bash-4.3/debian/patches/pgrp-pipe.diff bash-4.3/debian/patches/pgrp-pipe.diff --- bash-4.3/debian/patches/pgrp-pipe.diff 2013-10-23 14:41:22.0 +0200 +++ bash-4.3/debian/patches/pgrp-pipe.diff 2016-06-07 12:17:05.0 +0200 @@ -1,11 +1,43 @@ -# DP: Define PGRP_PIPE to avoid race condition. - a/config-bot.h -+++ b/config-bot.h -@@ -197,3 +197,6 @@ - - /* If you don't want bash to provide a default mail file to check. */ - #undef DEFAULT_MAIL_DIRECTORY -+ -+/* Bug #224543 */ -+#define PGRP_PIPE 1 +Description: Set PGRP_PIPE unconditionally on Linux, reproducibly + The original fix to #224543 involved defining this unconditionally in + config-bot.h. Unfortunately, upstream has a check in configure.ac that defines + this conditionally in config.h, which makes the bash-builtins package + unreproducible between different kernels. Here, we adopt a different
Bug#806945: bash: Please make bash build reproducibly
(Chet, your specific attention is required for this email, please) Ximin Luo: > On Sat, 28 May 2016 13:38:35 +0200 Reiner Herrmannwrote: >> After that, the only remaining issue is that the included header file >> /usr/include/bash/config.h varies depending on the kernel version used >> during build [1] (with kernel <4, PGRP_PIPE is defined). > > For this particular example, we can just patch this out, i.e. remove it from > the installed config.h. Debian already forces PGRP_PIPE 1 in config-bot.h, > which config.h includes at the end. > I dug into this a bit more and it looks like the cause of the difference is this snippet from configure.ac: linux*) LOCAL_LDFLAGS=-rdynamic # allow dynamic loading case "`uname -r`" in 2.[[456789]]*|3*) AC_DEFINE(PGRP_PIPE) ;; esac ;; This was added between bash-3.0.16 and bash-3.1, way before Linux 4 came out. So I wonder if this snippet should instead be: linux*) LOCAL_LDFLAGS=-rdynamic # allow dynamic loading case "`uname -r`" in 1.*|2.[[0123]]*) true ;; *) AC_DEFINE(PGRP_PIPE) ;; esac ;; to set this for all future kernels? Then Debian (and probably other distros) could get rid of our patch, too. However, the question still remains why config.h is installed into the end-user system, and if bash-built-with-linux-5 required PGRP_PIPE to be *undefined*, we would still have a reproducibility problem. Ximin -- GPG: ed25519/56034877E1F87C35 GPG: rsa4096/1318EFAC5FBBDBCE git://github.com/infinity0/pubkeys.git
Bug#806945: bash: Please make bash build reproducibly
CC'ing everyone that was on the previous conversation; Debian's BTS doesn't do this automatically :( We've made some progress on the Debian side; there's a few more stumbling blocks though: On Sat, 28 May 2016 13:38:35 +0200 Reiner Herrmannwrote: > Hi, > > it looks like most of the documentation related issues are now solved > by fixed toolchain packages. But it still uses the embedded and > outdated copy of man2html, which doesn't support SOURCE_DATE_EPOCH yet. > The attached patch uses the system man2html instead of the embedded one. > > After that, the only remaining issue is that the included header file > /usr/include/bash/config.h varies depending on the kernel version used > during build [1] (with kernel <4, PGRP_PIPE is defined). > For this particular example, we can just patch this out, i.e. remove it from the installed config.h. Debian already forces PGRP_PIPE 1 in config-bot.h, which config.h includes at the end. In general however, installing config.h is a code smell and an anti-pattern - it takes test results from the *build machine*, and then forces my machine to assume those. The presence of this anti-pattern potentially can make future versions unreproducible again, and we'll have to think of new ways to fix those, since this PGRP_PIPE forcing is just a lucky coincidence. The ideal solution from a software architecture viewpoint, would be (1) make the headers platform independent and not require a config.h, or if this is truly impossible then (2) instead of installing config.h, install some scripts to allow the user to generate their own config.h, with their own values. However, I don't know how much effort either of these options are. Only a few installed headers actually need config.h; perhaps these could be fixed to *not* require it: /usr/include/bash$ grep -r '#include .config.h.' builtins.h:#include "config.h" lib/glob/strmatch.h:#include shell.h:#include "config.h" shmbutil.h:#include X -- GPG: ed25519/56034877E1F87C35 GPG: rsa4096/1318EFAC5FBBDBCE git://github.com/infinity0/pubkeys.git
Bug#806945: bash: Please make bash build reproducibly
Hi, it looks like most of the documentation related issues are now solved by fixed toolchain packages. But it still uses the embedded and outdated copy of man2html, which doesn't support SOURCE_DATE_EPOCH yet. The attached patch uses the system man2html instead of the embedded one. After that, the only remaining issue is that the included header file /usr/include/bash/config.h varies depending on the kernel version used during build [1] (with kernel <4, PGRP_PIPE is defined). Regards, Reiner [1]: https://sources.debian.net/src/bash/4.3-14/configure.ac/#L1088 diff --git a/debian/control b/debian/control index b16a1c0..2e873fc 100644 --- a/debian/control +++ b/debian/control @@ -5,7 +5,7 @@ Maintainer: Matthias KloseStandards-Version: 3.9.6 Build-Depends: autoconf, autotools-dev, bison, libncurses5-dev, texinfo, texi2html, debhelper (>= 5), locales, gettext, sharutils, time, - xz-utils, dpkg-dev (>= 1.16.1) + xz-utils, dpkg-dev (>= 1.16.1), man2html Build-Depends-Indep: texlive-latex-base, ghostscript, texlive-fonts-recommended Homepage: http://tiswww.case.edu/php/chet/bash/bashtop.html Vcs-Browser: https://code.launchpad.net/~doko/+junk/pkg-bash-debian diff --git a/debian/rules b/debian/rules index 201eddf..3e78d6b 100755 --- a/debian/rules +++ b/debian/rules @@ -136,7 +136,7 @@ static-configure: bash-doc-build: stamps/stamp-build-bash-doc stamps/stamp-build-bash-doc: rm -f bash/doc/bashref.info - $(MAKE) -C build-bash/doc info html + $(MAKE) -C build-bash/doc info html MAN2HTML=/usr/bin/man2html $(MAKE) -C build-bash/doc bash.pdf bashref.pdf touch stamps/stamp-build-bash-doc signature.asc Description: PGP signature
Bug#806945: bash: Please make bash build reproducibly
Source: bash Version: 4.3-14 Severity: wishlist User: reproducible-bui...@lists.alioth.debian.org Usertags: timestamps Hi, While working on the “reproducible builds” effort [1], we have noticed that bash could not be built reproducibly. There are two problems: 1. Bash uses an embedded copy of man2html which produces html that contains timestamps, it is recommended to drop this internal copy and instead depend on the Debian man2html which contains a patch [4]. 2. The pdf files created by dvipdfmx contain fonts with indeterministic order and naming [2]. This can be fixed by not generating the pdf in the first place as gnu codding standards makes pdf generation optional [3]. This has to be solved upstream. Regards, akira [1]: https://wiki.debian.org/ReproducibleBuilds [2]: https://reproducible.debian.net/issues/unstable/fonts_in_pdf_files_issue.html [3]: https://www.gnu.org/prep/standards/standards.html#Standard-Targets [4]: http://sources.debian.net/src/man2html/1.6g-8/debian/patches/035-source-date-epoch.patch/