[Reproducible-builds] Bug#834861: cookiecutter: please make the build reproducible

[Chris Lamb]

Source: cookiecutter
Version: 1.4.0-2
Severity: wishlist
Tags: patch
User: reproducible-builds@lists.alioth.debian.org
Usertags: environment
X-Debbugs-Cc: reproducible-builds@lists.alioth.debian.org

Hi,

Whilst working on the Reproducible Builds effort [0], I noticed
that cookiecutter could not be built reproducibly.

Patch attached.

 [0] https://reproducible-builds.org/


Regards,

-- 
  ,''`.
 : :'  : Chris Lamb
 `. `'`  la...@debian.org / chris-lamb.co.uk
   `-
--- a/debian/patches/0003-Reproducible-build.patch  1970-01-01 
01:00:00.0 +0100
--- b/debian/patches/0003-Reproducible-build.patch  2016-08-19 
23:59:45.688697595 +0100
@@ -0,0 +1,54 @@
+Description: Make the build reproducible
+Author: Chris Lamb 
+Last-Update: 2016-08-19
+
+--- cookiecutter-1.4.0.orig/cookiecutter/config.py
 cookiecutter-1.4.0/cookiecutter/config.py
+@@ -22,6 +22,7 @@ from .exceptions import InvalidConfigura
+ 
+ logger = logging.getLogger(__name__)
+ 
++NOT_PROVIDED = object()
+ USER_CONFIG_PATH = os.path.expanduser('~/.cookiecutterrc')
+ 
+ DEFAULT_CONFIG = {
+@@ -68,7 +69,7 @@ def get_config(config_path):
+ return config_dict
+ 
+ 
+-def get_user_config(config_file=USER_CONFIG_PATH):
++def get_user_config(config_file=NOT_PROVIDED):
+ """Retrieve the config from a file or return the defaults if None is
+ passed. If an environment variable `COOKIECUTTER_CONFIG` is set up, try
+ to load its value. Otherwise fall back to a default file or config.
+@@ -77,6 +78,10 @@ def get_user_config(config_file=USER_CON
+ if config_file is None:
+ return copy.copy(DEFAULT_CONFIG)
+ 
++# Differentiate between being passed ``None`` and the default.
++if config_file is NOT_PROVIDED:
++config_file = USER_CONFIG_PATH
++
+ # Load the given config file
+ if config_file and config_file is not USER_CONFIG_PATH:
+ return get_config(config_file)
+--- cookiecutter-1.4.0.orig/cookiecutter/main.py
 cookiecutter-1.4.0/cookiecutter/main.py
+@@ -16,7 +16,7 @@ import logging
+ import os
+ import re
+ 
+-from .config import get_user_config, USER_CONFIG_PATH
++from .config import get_user_config, NOT_PROVIDED
+ from .exceptions import InvalidModeException, RepositoryNotFound
+ from .prompt import prompt_for_config
+ from .generate import generate_context, generate_files
+@@ -71,7 +71,7 @@ def expand_abbreviations(template, confi
+ def cookiecutter(
+ template, checkout=None, no_input=False, extra_context=None,
+ replay=False, overwrite_if_exists=False, output_dir='.',
+-config_file=USER_CONFIG_PATH):
++config_file=NOT_PROVIDED):
+ """
+ API equivalent to using Cookiecutter at the command line.
+ 
--- a/debian/patches/series 2016-08-19 23:41:14.643167381 +0100
--- b/debian/patches/series 2016-08-19 23:50:49.588100239 +0100
@@ -1,2 +1,3 @@
 0001-Don-t-test-for-.DS_Store.patch
 0002-Use-PyYAML-instead-of-poyo.patch
+0003-Reproducible-build.patch
___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

[Reproducible-builds] Bug#834859: echoping: please make the build reproducible

[Chris Lamb]

Source: echoping
Version: 6.0.2-9
Severity: wishlist
Tags: patch
User: reproducible-builds@lists.alioth.debian.org
Usertags: timestamps
X-Debbugs-Cc: reproducible-builds@lists.alioth.debian.org

Hi,

Whilst working on the Reproducible Builds effort [0], I noticed
that echoping could not be built reproducibly.

Patch attached.

 [0] https://reproducible-builds.org/


Regards,

-- 
  ,''`.
 : :'  : Chris Lamb
 `. `'`  la...@debian.org / chris-lamb.co.uk
   `-
--- a/debian/patches/006_reproducible-build.diff1970-01-01 
01:00:00.0 +0100
--- b/debian/patches/006_reproducible-build.diff2016-08-19 
23:25:14.950925873 +0100
@@ -0,0 +1,40 @@
+Description: Make the build reproducible
+Author: Chris Lamb 
+Last-Update: 2016-08-19
+
+--- echoping-6.0.2.orig/configure
 echoping-6.0.2/configure
+@@ -23320,8 +23320,13 @@ echo "${ECHO_T}disabled" >&6; }
+ fi
+ 
+ 
+-compil_date=`date +%Y-%m-%d`
+-hostname=$ac_hostname
++if test "x$SOURCE_DATE_EPOCH" = "x"; then
++  compil_date=`date +%Y-%m-%d`
++  hostname=$ac_hostname
++else
++  compil_date=`date --utc --date="@$SOURCE_DATE_EPOCH" +%Y-%m-%d`
++  hostname=generic
++fi
+ 
+ 
+ 
+--- echoping-6.0.2.orig/configure.ac
 echoping-6.0.2/configure.ac
+@@ -334,8 +334,13 @@ DISPLAY_SETTING(LIBIDN)
+ DISPLAY_SETTING(TOS)
+ DISPLAY_SETTING(PRIORITY)
+ 
+-compil_date=`date +%Y-%m-%d`
+-hostname=$ac_hostname
++if test "x$SOURCE_DATE_EPOCH" = "x"; then
++  compil_date=`date +%Y-%m-%d`
++  hostname=$ac_hostname
++else
++  compil_date=`date --utc --date="@$SOURCE_DATE_EPOCH" +%Y-%m-%d`
++  hostname=generic
++fi
+ AC_SUBST(hostname)
+ AC_SUBST(compil_options)
+ AC_SUBST(compil_date)
--- a/debian/patches/series 2016-08-19 23:13:25.240754904 +0100
--- b/debian/patches/series 2016-08-19 23:16:12.530211168 +0100
@@ -3,3 +3,4 @@
 003-fix-for-https-creashes.diff
 002-FTBFS_against_gnutls26_greater_the_2.7.x_fix.diff
 001-manpage_fixes.diff
+006_reproducible-build.diff
___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

[Reproducible-builds] Bug#834857: nagios-nrpe: please make the build reproducible

[Chris Lamb]

Source: nagios-nrpe
Version: 2.15-1
Severity: wishlist
Tags: patch
User: reproducible-builds@lists.alioth.debian.org
Usertags: timestamps randomness
X-Debbugs-Cc: reproducible-builds@lists.alioth.debian.org

Hi,

Whilst working on the Reproducible Builds effort [0], I noticed
that nagios-nrpe could not be built reproducibly.

Whilst I "fix" the Diffie-Hellman key parameters, this is no worse
than the current situation in that they were random across builds:
everyone using the (for example) amd64 package was using the same
parameters anyway…

Patch attached.

 [0] https://reproducible-builds.org/


Regards,

-- 
  ,''`.
 : :'  : Chris Lamb
 `. `'`  la...@debian.org / chris-lamb.co.uk
   `-
--- a/debian/patches/00list 2016-08-19 21:13:52.279707711 +0100
--- b/debian/patches/00list 2016-08-19 22:32:45.467992041 +0100
@@ -4,3 +4,4 @@
 06_pid_directory.dpatch
 07_warn_ssloption.dpatch
 09_noremove_pid.dpatch
+10_reproducible_build.dpatch
--- a/debian/patches/10_reproducible_build.dpatch   1970-01-01 
01:00:00.0 +0100
--- b/debian/patches/10_reproducible_build.dpatch   2016-08-19 
22:37:32.874466180 +0100
@@ -0,0 +1,26 @@
+#! /bin/sh /usr/share/dpatch/dpatch-run
+## 10_reproducible_build.dpatch by Chris Lamb 
+##
+## All lines beginning with `## DP:' are a description of the patch.
+## DP: Make the build reproducible.
+
+@DPATCH@
+diff -urNad '--exclude=CVS' '--exclude=.svn' '--exclude=.git' 
'--exclude=.arch' '--exclude=.hg' '--exclude=_darcs' '--exclude=.bzr' 
pkg-nrpe~/update-version pkg-nrpe/update-version
+--- pkg-nrpe~/update-version   2016-08-19 21:13:52.279707711 +0100
 pkg-nrpe/update-version2016-08-19 22:29:19.434217944 +0100
+@@ -20,11 +20,11 @@
+ 
+ # Get date (two formats)
+ if [ -n "$2" ]; then
+-LONGDATE=`date -d "$2" "+%B %d, %Y"`
+-SHORTDATE=`date -d "$2" "+%m-%d-%Y"`
++LONGDATE=$(LC_ALL=C date -u -d "$2" "+%B %d, %Y")
++SHORTDATE=$(date -u -d "$2" "+%m-%d-%Y")
+ else
+-LONGDATE=`date "+%B %d, %Y"`
+-SHORTDATE=`date "+%m-%d-%Y"`
++LONGDATE=$(LC_ALL=C date -u -d "@${SOURCE_DATE_EPOCH:-$(date +%s)}" "+%B 
%d, %Y")
++SHORTDATE=$(date -u -d "@${SOURCE_DATE_EPOCH:-$(date +%s)}" "+%m-%d-%Y")
+ fi
+ 
+ # Current version number
--- a/debian/rules  2016-08-19 21:13:52.279707711 +0100
--- b/debian/rules  2016-08-19 22:52:57.430353150 +0100
@@ -10,6 +10,8 @@
dh $@ --with dpatch,autotools_dev
 
 override_dh_auto_configure:
+   # Save deterministic "openssl dhparam" output.
+   cp include/dh.h include/dh.h.orig
./configure \
--prefix=/usr \
--enable-ssl \
@@ -18,5 +20,7 @@
--localstatedir=/var \
--libexecdir=/usr/lib/nagios/plugins \
--libdir=/usr/lib/nagios
+   # Restore deterministic "openssl dhparam" output.
+   cp include/dh.h.orig include/dh.h
 
 override_dh_auto_install:
___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

Re: [Reproducible-builds] Reproducible patches for libisoburn and libisofs

[Thomas Schmitt]

Hi,

i wrote:
> > As it is now, it is the clearest no-brainer:
> > Export SOURCE_DATE_EPOCH and forget about any other time issues which
> > you don't create yourself by own xorriso arguments.

Chris Lamb wrote:
> I'm sorry but I don't quite follow and I don't want to read what I want
> to read.

Currently it is (hopefully) this way:

If the same SOURCE_DATE_EPOCH value is in effect with each xorriso run
then the user may expect that timestamps in the ISO are not an obstacle
for reproducibility. Regardless of input.
So it is only about names, ownership, permissions, and data content.
With popular option -r one can make ownership and permissions flatly
reproducible.

This can be changed by own xorriso arguments to the three xorrisofs
options named in the man page text:
  --modification-date=, --gpt_disk_guid, --set_all_file_dates.


> trying to avoid the case of — heaven forbid we broke
> the Debian CD creation! — that we would get the blame as ACKing on their
> behalf. :)

The popular debian-cd ones i can ACK quite well myself. (And then i'd
first need to convince Steve McIntyre of using Sid's xorriso.)
I test with
  debian-8.4.0-amd64-netinst.iso
  debian-8.5.0-i386-lxde-CD-1.iso
  debian-live-8.4.0-amd64-standard.iso
  debian-8.1.0-arm64-netinst.iso
  debian-7.9.0-kfreebsd-amd64-netinst.iso
  debian-7.4.0-mips-netinst.iso
  debian-7.4.0-sparc-netinst.iso
  debian-7.0-hppa-NETINST-1.iso
  debian-8.3.0-ppc64el-netinst.iso
  debian-9.0-sparc64-NETINST-1.iso
and mini.iso for amd64 and i386.

I mount them, let xorriso pack up the files and replay the boot related
-as mkisofs options which it deduced from inspecting the unmounted ISO.

Then i let xorriso report both boot equipments and let diff compare the
reports. After mounting both ISOs, two find runs over both trees make
sure that differences in file content, ownership, permissions, or time
get reported.
The few reported differences come from automatic timestamps and differing
block addresses which get patched into boot images. (The originally used
xorriso versions still produced the reproducibility-unfriendly sequence
of file data extents.)

My scruples are more towards adventurous distros which immediately use
my newest releases.


Have a nice day :)

Thomas


___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

Re: [Reproducible-builds] ImportError: No module named pep8

[Mattia Rizzolo]

On Fri, Aug 19, 2016 at 05:54:57PM +0100, Chris Lamb wrote:
> Some extra background — we usually remove archived bugs from notes. I
> believe there is a script for this..

Just ran it.


I usually run it every so often, when I remember :)

-- 
regards,
Mattia Rizzolo

GPG Key: 66AE 2B4A FCCF 3F52 DA18  4D18 4B04 3FCD B944 4540  .''`.
more about me:  https://mapreri.org : :'  :
Launchpad user: https://launchpad.net/~mapreri  `. `'`
Debian QA page: https://qa.debian.org/developer.php?login=mattia  `-


signature.asc
Description: PGP signature
___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

Re: [Reproducible-builds] ImportError: No module named pep8

[Chris Lamb]

> So either the bug is here again, or it was not the real reason for the
> FTBFS

It's unclear to me as AIUI "pep8" is being renamed to pycodestyle, so it
could be that "neutron/hacking/checks.py" is at fault for importing pep8.
I'd just file a bug and see what the maintainer thinks, it's not really
something Reproducible-specific…

Some extra background — we usually remove archived bugs from notes. I
believe there is a script for this..


Regards,

-- 
  ,''`.
 : :'  : Chris Lamb
 `. `'`  la...@debian.org / chris-lamb.co.uk
   `-

___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

[Reproducible-builds] ImportError: No module named pep8

[Santiago Vila]

Greetings.

In notes, Bug#816735 appears in a lot of places as a reason for FTBFS
bugs like this:

 ImportError: No module named pep8

The bug is now closed and archived, but many of the packages having this
bug in notes happen to FTBFS again. Example:

https://tests.reproducible-builds.org/debian/rb-pkg/unstable/amd64/neutron.html

So either the bug is here again, or it was not the real reason for the
FTBFS (I assume we don't put a bug number in a package in notes only
because the package has a similar bug).

Can anyone confirm / clarify this?

Thanks.

___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

Re: [Reproducible-builds] Reproducible patches for libisoburn and libisofs

[Chris Lamb]

> > *iff* SOURCE_DATE_EPOCH is set:
> > 
> >  - mtimes are taken from stat(2). [The FS creator is responsible for
> >   ensuring they are reproducible, possibly by some "clamping" call
> >   to `find -newermt | xargs touch …`.]
> >
> > - atime is copied from mtime [as FS creator "can't" set that].
> >
> > - ctime is copied from mtime [ditto].
> 
> man 2 utime promises to set atime and xorriso tries to restore it when
> extracting files to disk.

I put "can't" in quotes as there is no pervasive/widespread shell-level
tool for this (cf. touch(1)) so I stand by this.

> But for ctime i know no portable method.

Mm.

> Export SOURCE_DATE_EPOCH and forget about any other time issues which
> you don't create yourself by own xorriso arguments.

I'm sorry but I don't quite follow and I don't want to read what I want
to read. Please could you clarify exactly the behaviour in this case?

> Looks like  --set_all_file_dates  is losing its only user even before it
> gets released.

(Which is good; the less options the better IMHO.)

> > I am therefore not testing them from
> > the "Debian" point of view and ACKs are simply my own testcases.
> 
> Any ACK or NAK is highly welcome.

I'm sure; I was just trying to avoid the case of — heaven forbid we broke
the Debian CD creation! — that we would get the blame as ACKing on their
behalf. :)


Regards,

-- 
  ,''`.
 : :'  : Chris Lamb
 `. `'`  la...@debian.org / chris-lamb.co.uk
   `-

___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

Re: [Reproducible-builds] Reproducible patches for libisoburn and libisofs

[Thomas Schmitt]

Hi,

Chris Lamb wrote:
> May I propose the following behaviour *iff* SOURCE_DATE_EPOCH
> is set:
> 
>  - mtimes are taken from stat(2). [The FS creator is responsible for
>   ensuring they are reproducible, possibly by some "clamping" call
>   to `find -newermt | xargs touch …`.]
>
> - atime is copied from mtime [as FS creator "can't" set that].
>
> - ctime is copied from mtime [ditto].

man 2 utime promises to set atime and xorriso tries to restore it when
extracting files to disk. But for ctime i know no portable method.
One needs to be filesystem ultra user to influence that one.
(E.g. operator of xorriso inside the emerging ISO.)


Looks like  --set_all_file_dates  is losing its only user even before it
gets released.

Consider:
Your new FS creator responsibilities dilute the radical decision to which
the crowd pushed me a few days ago. As it is now, it is the clearest
no-brainer:
Export SOURCE_DATE_EPOCH and forget about any other time issues which
you don't create yourself by own xorriso arguments.

We can justify this behavior by pointing to program cp which, too, updates
all file times by default. Just that we set and hold the clock as we desire.


Ok. It is definitely too early for release. :))
(I understand we have time until end of september for Stretch ...)


> This seems less like a giant hammer of setting them all to target->now.

Until release we can reshape it to a screwdriver.
Afterwards i am restricted by general compatibility commitments.


I need to think where and how to implement the new proposal.
Probably it is worth regardless how SOURCE_DATE_EPOCH will finally
be interpreted.

If it becomes an alternative to --set_all_file_dates then the user is
free to override a hammer SOURCE_DATE_EPOCH by the new optiom
--set_file_times_to_mtime (is this a good name ?).


> I am therefore not testing them from
> the "Debian" point of view and ACKs are simply my own testcases.

Any ACK or NAK is highly welcome.


Have a nice day :)

Thomas


___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

Re: [Reproducible-builds] enable build path variation when testing unstable & experimental?

[HW42]

Chris Lamb:
> Holger wrote:
> 
>> I think regarding Debian releases, I'd recommend that we aim for
>> partially reproducible packages given known build paths
> 
> I agree. We should, regrettably, define a fixed build path for stretch
> at this point in the release cycle.

I want to note that this does not require any direct changes (when dpkg
in stretch has .buildinfo support). The build path is going to be
recorded in the .builinfo anyway and sbuild already
has a --build-path option.

So this is only relevant as a) internal definition b) how we are going
to test packages from the stretch archive.

> I do not believe we would have enough resources to fix the issues that
> would arise in time to make the case for a reproducible stretch as it
> would, of course, need to be made well before the release date itself.
> 
>> shall we wait til testing/i386 has been built once fully with build
>> path variation so we *know* the number I currently guesstimate as 75-80%??

I don't care either way.

> Whilst I'm curious, knowing the number makes no difference to me; I
> would recommend the same course of action whether it was "too low" or
> "fine". This also avoids a subjective debate on which these terms
> would apply to the number.
> 
>> shall we enable build path variation for unstable and experimental
>> for all tested archs now?
> 
> Yes, please go ahead as this reflects our goals for stretch+1. I vehemently
> believe our numbers should match our goals as close as reasonably possible.

+1



signature.asc
Description: OpenPGP digital signature
___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

Re: [Reproducible-builds] Reproducible patches for libisoburn and libisofs

[Chris Lamb]

Hi Thomas,

> - Verify that SOURCE_DATE_EPOCH works as promised and expected.
>   (Promised is to set the defaults of three xorriso settings.
>Expected is that these defaults suffice for reliable reproducibility.)

My current schedule is to test this and report back in the next 4 or 5
days.

As a heads-up, I think I am likely to relax one change that might be
of interest. May I propose the following behaviour *iff* SOURCE_DATE_EPOCH
is set:

 - mtimes are taken from stat(2). [The FS creator is responsible for
   ensuring they are reproducible, possibly by some "clamping" call
   to `find -newermt | xargs touch …`.]

 - atime is copied from mtime [as FS creator "can't" set that].

 - ctime is copied from mtime [ditto].

This seems less like a giant hammer of setting them all to target->now.

 
> - Verify that bootable Debian ISOs made with xorriso-1.4.6 still work
>   as good or bad as those currently made with 1.3.6 (or maybe 1.4.4).

To clear up any ambiguity here, whilst I am making ISOs based on Debian,
they are not "Debian ISOs" in any sense of being connected with debian-cd
or being official in any sense. I am therefore not testing them from
the "Debian" point of view and ACKs are simply my own testcases.


Regards,

-- 
  ,''`.
 : :'  : Chris Lamb
 `. `'`  la...@debian.org / chris-lamb.co.uk
   `-

___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

Re: [Reproducible-builds] automated diffoscope for parallel build bugs

[Holger Levsen]

On Fri, Aug 19, 2016 at 01:10:53PM +0100, Chris Lamb wrote:
> Only from me asking:
>  | So, just to be 100% clear, simply varying DEB_BUILD_OPTIONS="parallel=X"
>  | would not have discovered this gedit bug? 
> Where — in context — varying meant 17 vs. 18 instead of 1 vs. 18.

it wasn't clear to me that the resulting gedit package would be
reproducible. in fact, gedit with non parallel build currently still
aint reproducible on i386+armhf, maybe thats why.
 
> (Besides, we've seen enough strange packages to think that anything is
> possible…)

right. "so rare that's it's hardly worth testing and adding complexity…"


-- 
cheers,
Holger


signature.asc
Description: Digital signature
___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

Re: [Reproducible-builds] Reproducible patches for libisoburn and libisofs

[Thomas Schmitt]

Hi,

i am pondering when to release xorriso-1.4.6 et.al.
1.4.5 passed my rebuild-ISO regression tests by reporting no differences
of file content or boot equipment as far as xorriso can replay the latter.

Normally i would wait a few weeks whether my daily backups or some
courageous user find any other problems.
But my backups hardly touch the code parts which underwent the dangerous
changes. Courageous bootability users are sparse.

So, if i release 1.4.6 now and get it into Debian Sid a few days later,
do you already have test cases where the new code would be challenged ?

- Verify that SOURCE_DATE_EPOCH works as promised and expected.
  (Promised is to set the defaults of three xorriso settings.
   Expected is that these defaults suffice for reliable reproducibility.)

- Verify that bootable Debian ISOs made with xorriso-1.4.6 still work
  as good or bad as those currently made with 1.3.6 (or maybe 1.4.4).


Have a nice day :)

Thomas


___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

Re: [Reproducible-builds] automated diffoscope for parallel build bugs

[Chris Lamb]

> > This entire email chain was prompted by such a case.
> 
> what makes you think so? this wasn't and isn't clear for me, neither
> from the mails nor from https://bugzilla.gnome.org/show_bug.cgi?id=770100 

Only from me asking:

 | So, just to be 100% clear, simply varying DEB_BUILD_OPTIONS="parallel=X"
 | would not have discovered this gedit bug? 

Where — in context — varying meant 17 vs. 18 instead of 1 vs. 18.

*shrugs*

(Besides, we've seen enough strange packages to think that anything is
possible…)


Regards,

-- 
  ,''`.
 : :'  : Chris Lamb
 `. `'`  la...@debian.org / chris-lamb.co.uk
   `-

___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

Re: [Reproducible-builds] Remaining reprotest variations

[Chris Lamb]

Hi Ceridwen,

> For most of the variations I've done so far, I've been either
> depending on external utilities or had POSIX-compliant ways to execute
> them.  The rest of the variations pose more problems.

[…]

I'm afraid I didn't get around to replying to this at the time and I'm
sure you have resolved most — if not all — of these issues, so to respond
to them might just be confusing at this point.

However, I did want to comment generally that this kind of mail shows you
have been and will continue to be an asset to the Reproducible Builds
community.


Best wishes,

-- 
  ,''`.
 : :'  : Chris Lamb
 `. `'`  la...@debian.org / chris-lamb.co.uk
   `-

___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

Re: [Reproducible-builds] automated diffoscope for parallel build bugs

[Holger Levsen]

On Fri, Aug 19, 2016 at 12:32:13PM +0100, Chris Lamb wrote:
> > yes, I cannot imagine there's something which is reproducible if build
> > in parallel but not if not.
> This entire email chain was prompted by such a case.

what makes you think so? this wasn't and isn't clear for me, neither
from the mails nor from https://bugzilla.gnome.org/show_bug.cgi?id=770100 


-- 
cheers,
Holger


signature.asc
Description: Digital signature
___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

Re: [Reproducible-builds] enable build path variation when testing unstable & experimental?

[Chris Lamb]

Holger wrote:

> I think regarding Debian releases, I'd recommend that we aim for
> partially reproducible packages given known build paths

I agree. We should, regrettably, define a fixed build path for stretch
at this point in the release cycle.

I do not believe we would have enough resources to fix the issues that
would arise in time to make the case for a reproducible stretch as it
would, of course, need to be made well before the release date itself.

> shall we wait til testing/i386 has been built once fully with build
> path variation so we *know* the number I currently guesstimate as 75-80%??

Whilst I'm curious, knowing the number makes no difference to me; I
would recommend the same course of action whether it was "too low" or
"fine". This also avoids a subjective debate on which these terms
would apply to the number.

> shall we enable build path variation for unstable and experimental
> for all tested archs now?

Yes, please go ahead as this reflects our goals for stretch+1. I vehemently
believe our numbers should match our goals as close as reasonably possible.


Regards,

-- 
  ,''`.
 : :'  : Chris Lamb
 `. `'`  la...@debian.org / chris-lamb.co.uk
   `-

___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

Re: [Reproducible-builds] automated diffoscope for parallel build bugs

[Chris Lamb]

> > Right, so make all the pkg-gnome packages build reproducibly without
> > paralellism enabled (which you should do anyway) and then simply
> > build with parallel enabled.
> 
> yes, I cannot imagine there's something which is reproducible if build
> in parallel but not if not.

This entire email chain was prompted by such a case.


Regards,

-- 
  ,''`.
 : :'  : Chris Lamb
 `. `'`  la...@debian.org / chris-lamb.co.uk
   `-

___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

Re: [Reproducible-builds] automated diffoscope for parallel build bugs

[Holger Levsen]

Hi,

On Fri, Aug 19, 2016 at 12:58:24AM +0100, Chris Lamb wrote:
> > I understand that building with parallel disabled takes much longer
> > for many packages so I don't know if this is just a lack of resources
> It's more that we have two builds and I think (?) we would prefer to do
> different values of N > 1. So a different kind of resource problem.

this and there are more issues:

- quite a lot of packages force parallel builds in some way or another
  in their build systems, so these wouldnt be tested in this regard
- this aint a real world scenario for our use case, which is testing and
  working on reproducible builds. IOW: this is just another area of QA
  work, which I agree should probably be done, but it's out of scope for
  our project and doing it would draw "human ressources".
- actual tests would run a *a lot* slower, thus we would see the results
  were are interested in, at a later time, even more disconnected from
  the actual upload
- the overall results would become older

> It wouldn't be too much work to run such an test yourself or to use
> our infrastructure.

it's actually rather easy to build all packages in a loop… seriously, I
mean:

cd $path_to_svn
for i in $(find * -type d) ; do 
cd $i && debuild && cd ..
done

now you will have a lot of debs :) copy them away, apply whatever means
to enable or disable parallelism and run that loop again.

now do another loop, diff each pair of debs and if they are the same,
delete them. investigate the rest using diffoscope :)
 
> The problem is all about how you use the results; in the case where a
> package differed between builds you would not know whether this was
> due to the parallelism that was introduced or whether the package
> is inherently non-deterministic.

yup, but 
https://tests.reproducible-builds.org/debian/unstable/amd64/pkg_set_gnome.html
suggests most of the core gnome packages are actually reproducible
today…

> > Building everything in pkg-gnome svn individually with and without
> > parallel and then running diffoscope by hand to compare every binary
> > package is not much fun.
> 
> Right, so make all the pkg-gnome packages build reproducibly without
> paralellism enabled (which you should do anyway) and then simply
> build with parallel enabled.

yes, I cannot imagine there's something which is reproducible if build
in parallel but not if not.

So indeed, making packages build reproducible when build in parallel
will fix your original issue as well! :)


-- 
cheers,
Holger


signature.asc
Description: Digital signature
___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds