Bug#783210: glibc: please make the package build reproducibly

[Holger Levsen]

On Mon, Dec 19, 2016 at 10:11:00AM +, Ximin Luo wrote:
> It is not strictly necessary, but it allows one to build reproducibly even 
> when not using dpkg-buildpackage, e.g. when trying to re-run a specific part 
> of the build via `debian/rules build` or something else. This is helpful to 
> save time when debugging.

using dpkg-buildpackage is also not mandatory, eg developers could build
with "./debian/rules binary" initially and upload this to the archive…


-- 
cheers,
Holger

(and yes, we need to get rid off developer built uploads, _except_ for
bootstrapping only… amen.)


signature.asc
Description: Digital signature

Bug#783210: glibc: please make the package build reproducibly

[Ximin Luo]

Aurelien Jarno:
> [..]
> 
> It would be nice to have an explanation why the changes from your patch
> are needed. See my comments below.
> 
> 
>> diff -Nru glibc-2.24/debian/rules glibc-2.24/debian/rules
>> --- glibc-2.24/debian/rules  2016-11-25 21:59:04.0 +
>> +++ glibc-2.24/debian/rules  2016-11-15 18:03:37.0 +
>> @@ -45,6 +45,7 @@
>>  
>>  DEB_SOURCE_PACKAGE := $(strip $(shell egrep '^Source: ' debian/control | 
>> cut -f 2 -d ':'))
>>  
>> +SOURCE_DATE_EPOCH ?= $(shell dpkg-parsechangelog -STimestamp)
>>  DEB_VERSION := $(shell dpkg-parsechangelog | egrep '^Version:' | cut -f 2 
>> -d ' ')
>>  GLIBC_VERSION = $(shell echo $(DEB_VERSION) | sed -e 's/.*://' -e 
>> 's/[+-].*//')
> 
> You told above that SOURCE_DATE_EPOCH is already set by
> dpkg-buildpackage, so why do we need to do that?
> 

It is not strictly necessary, but it allows one to build reproducibly even when 
not using dpkg-buildpackage, e.g. when trying to re-run a specific part of the 
build via `debian/rules build` or something else. This is helpful to save time 
when debugging.

>> diff -Nru glibc-2.24/debian/rules.d/build.mk 
>> glibc-2.24/debian/rules.d/build.mk
>> --- glibc-2.24/debian/rules.d/build.mk   2016-11-25 12:02:24.0 
>> +
>> +++ glibc-2.24/debian/rules.d/build.mk   2016-11-15 18:03:37.0 
>> +
>> @@ -316,18 +316,16 @@
>>  $(stamp)source: $(stamp)patch
>>  mkdir -p $(build-tree)
>>  cd .. && \
>> -   find $(GLIBC_SOURCES) -depth -newermt '$(DEB_BUILD_DATE)' \
>> --print0 | \
>> -   xargs -0r touch --no-dereference --date='$(DEB_BUILD_DATE)'
>> -cd .. && \
>>  find $(GLIBC_SOURCES) -print0 | \
>>  LC_ALL=C sort -z | \
>>  tar -c -J --null -T - --no-recursion \
>>  --mode=go=rX,u+rw,a-s \
>> +--clamp-mtime --mtime "@$(SOURCE_DATE_EPOCH)" \
>>  --owner=root --group=root --numeric-owner \
>>  -f $(CURDIR)/$(build-tree)/glibc-$(GLIBC_VERSION).tar.xz
>>  mkdir -p debian/glibc-source/usr/src/glibc
>>  tar cf - --files-from debian/glibc-source.filelist \
>> +--clamp-mtime --mtime "@$(SOURCE_DATE_EPOCH)" \
>>| tar -x -C debian/glibc-source/usr/src/glibc -f -
> 
> Why do we need to change this, what doesn't work in the current code? The 
> current code
> has been provided by Jérémy Bobbio in the current bug.
> 

Yes, it doesn't work. but also it was not applied correctly:

1. The original patch had a mistake in: we need "@$(unix_timestamp)", the '@' 
tells GNU find/tar/date/etc to interpret the subsequent value as a UNIX 
timestamp. Otherwise these tools will complain about an invalid date, but 
silently use a different value.

2. The original patch was not applied fully/correctly either - nowhere in the 
current glibc packaging is DEB_BUILD_DATE actually set. The original patch set 
this, instead of my previous hunk that sets SOURCE_DATE_EPOCH, however this is 
not present in the current glibc packaging.

3. Since the original patch was written, we got the --clamp-mtime feature into 
GNU tar, and this allows us to make the patch smaller and the resulting code is 
neater. (It also avoids writing directly to the filesystem.)

>>  touch $@
>> diff -Nru glibc-2.24/debian/rules.d/debhelper.mk 
>> glibc-2.24/debian/rules.d/debhelper.mk
>> --- glibc-2.24/debian/rules.d/debhelper.mk   2016-11-25 22:08:30.0 
>> +
>> +++ glibc-2.24/debian/rules.d/debhelper.mk   2016-11-15 18:03:37.0 
>> +
>> @@ -77,8 +77,7 @@
>>  -exec chmod a+x '{}' ';'
>>  dh_makeshlibs -Xgconv/ -p$(curpass) -V "$(call xx,shlib_dep)"
>>  # Add relevant udeb: lines in shlibs files
>> -chmod a+x debian/shlibs-add-udebs
>> -./debian/shlibs-add-udebs $(curpass)
>> +sh ./debian/shlibs-add-udebs $(curpass)
>>  
>>  dh_installdeb -p$(curpass)
>>  dh_shlibdeps -p$(curpass)
> 
> Why do we need this change?
> 

This is not strictly necessary, but it helps one perform reproductions in the 
same source tree, by running dpkg-buildpackage twice. Otherwise, glibc-source 
has -x for that file in the first build and +x for that file in the second 
build.

Yes, our testing infrastructure does not run builds twice in the same source 
tree and neither does debrepro nor reprotest, however doing this is useful for 
debugging purposes, to save time and avoid having to re-run the whole build 
againg.

X

-- 
GPG: ed25519/56034877E1F87C35
GPG: rsa4096/1318EFAC5FBBDBCE
https://github.com/infinity0/pubkeys.git

Bug#783210: glibc: please make the package build reproducibly

[Aurelien Jarno]

On 2016-12-09 18:29, Ximin Luo wrote:
> - the upstream folks
> 
> Aurelien Jarno:
> > On 2016-06-06 02:48, Ximin Luo wrote:
> >> On Mon, 19 Oct 2015 10:11:25 +0200 Aurelien Jarno  
> >> wrote:
> >>> It means that the point 3 (usage of __DATE__ and __TIME__) is the only
> >>> one left to address in the version currently in experimental.
> >>>
> >>
> >> Hi, GCC have now added support for SOURCE_DATE_EPOCH so we can use this 
> >> instead. Debhelper now sets this environment variable automatically, and 
> >> GCC should automatically read it, so perhaps we can get rid of the whole 
> >> (3)-related patch, but I haven't tested this yet.
> > 
> > In the current code __DATE__ and __TIME__ are used to generate a unique
> > serial number shared between two processes. Therefore using
> > SOURCE_DATE_EPOCH for that is not correct here.
> > 
> > I have submitted a patch upstream to fix that another way [1].
> > Unfortunately there is still work to do and I am currently lacking time
> > to work on that. Help would be appreciated.
> > 
> 
> Hi Aurelian, I noticed that due to GCC supporting SOURCE_DATE_EPOCH and 
> dpkg-buildpackage automatically setting this now, glibc is actually already 
> reproducible [1] if you apply this patch. I've attached an updated version of 
> it in this email, for convenience. (It's the same as what I attached to [1].)
>
> The issue you mentioned, of this "unique id", is *already* a problem in the 
> currently-released version (in Debian testing and unstable) because of the 
> GCC+dpkg-buildpackage changes. So I think it doesn't make much sense to keep 
> holding this patch off now.

I don't get it. On one side you says that dpkg-buildpackage is already
exporting SOURCE_DATE_EPOCH so that the bug is already fixed. Then you
said that's the reason why the patch should be apply to fix the bug.

Why do we need to export SOURCE_DATE_EPOCH if it is already done by
dpkg-buildpackage?

> The concrete benefit of applying this patch, is that tests.r-b.org would be 
> able to produce sensible diffoscope output for glibc, so we could track both 
> [1] and any future cases of unreproducibility. Currently it times out for 
> glibc, because there are too many differences, due to the patch not being 
> applied.

I fail to understand why it prevents to have sensible diffoscope. The
current issue only concerns one binary in a single package. The other
parts of the provided patch have been applied more than one year ago.

> What do you think?

It would be nice to have an explanation why the changes from your patch
are needed. See my comments below.


> diff -Nru glibc-2.24/debian/rules glibc-2.24/debian/rules
> --- glibc-2.24/debian/rules   2016-11-25 21:59:04.0 +
> +++ glibc-2.24/debian/rules   2016-11-15 18:03:37.0 +
> @@ -45,6 +45,7 @@
>  
>  DEB_SOURCE_PACKAGE := $(strip $(shell egrep '^Source: ' debian/control | cut 
> -f 2 -d ':'))
>  
> +SOURCE_DATE_EPOCH ?= $(shell dpkg-parsechangelog -STimestamp)
>  DEB_VERSION := $(shell dpkg-parsechangelog | egrep '^Version:' | cut -f 2 -d 
> ' ')
>  GLIBC_VERSION = $(shell echo $(DEB_VERSION) | sed -e 's/.*://' -e 
> 's/[+-].*//')

You told above that SOURCE_DATE_EPOCH is already set by
dpkg-buildpackage, so why do we need to do that?


> diff -Nru glibc-2.24/debian/rules.d/build.mk 
> glibc-2.24/debian/rules.d/build.mk
> --- glibc-2.24/debian/rules.d/build.mk2016-11-25 12:02:24.0 
> +
> +++ glibc-2.24/debian/rules.d/build.mk2016-11-15 18:03:37.0 
> +
> @@ -316,18 +316,16 @@
>  $(stamp)source: $(stamp)patch
>   mkdir -p $(build-tree)
>   cd .. && \
> -find $(GLIBC_SOURCES) -depth -newermt '$(DEB_BUILD_DATE)' \
> - -print0 | \
> -   xargs -0r touch --no-dereference --date='$(DEB_BUILD_DATE)'
> - cd .. && \
>   find $(GLIBC_SOURCES) -print0 | \
>   LC_ALL=C sort -z | \
>   tar -c -J --null -T - --no-recursion \
>   --mode=go=rX,u+rw,a-s \
> + --clamp-mtime --mtime "@$(SOURCE_DATE_EPOCH)" \
>   --owner=root --group=root --numeric-owner \
>   -f $(CURDIR)/$(build-tree)/glibc-$(GLIBC_VERSION).tar.xz
>   mkdir -p debian/glibc-source/usr/src/glibc
>   tar cf - --files-from debian/glibc-source.filelist \
> + --clamp-mtime --mtime "@$(SOURCE_DATE_EPOCH)" \
> | tar -x -C debian/glibc-source/usr/src/glibc -f -

Why do we need to change this, what doesn't work in the current code? The 
current code
has been provided by Jérémy Bobbio in the current bug.

>   touch $@
> diff -Nru glibc-2.24/debian/rules.d/debhelper.mk 
> glibc-2.24/debian/rules.d/debhelper.mk
> --- glibc-2.24/debian/rules.d/debhelper.mk2016-11-25 22:08:30.0 
> +
> +++ glibc-2.24/debian/rules.d/debhelper.mk2016-11-15 18:03:37.0 
> +
> @@ -77,8 +77,7 @@
>   -exec chmod a+x '{}' ';'
>   

Bug#783210: glibc: please make the package build reproducibly

[Ximin Luo]

- the upstream folks

Aurelien Jarno:
> On 2016-06-06 02:48, Ximin Luo wrote:
>> On Mon, 19 Oct 2015 10:11:25 +0200 Aurelien Jarno  
>> wrote:
>>> It means that the point 3 (usage of __DATE__ and __TIME__) is the only
>>> one left to address in the version currently in experimental.
>>>
>>
>> Hi, GCC have now added support for SOURCE_DATE_EPOCH so we can use this 
>> instead. Debhelper now sets this environment variable automatically, and GCC 
>> should automatically read it, so perhaps we can get rid of the whole 
>> (3)-related patch, but I haven't tested this yet.
> 
> In the current code __DATE__ and __TIME__ are used to generate a unique
> serial number shared between two processes. Therefore using
> SOURCE_DATE_EPOCH for that is not correct here.
> 
> I have submitted a patch upstream to fix that another way [1].
> Unfortunately there is still work to do and I am currently lacking time
> to work on that. Help would be appreciated.
> 

Hi Aurelian, I noticed that due to GCC supporting SOURCE_DATE_EPOCH and 
dpkg-buildpackage automatically setting this now, glibc is actually already 
reproducible [1] if you apply this patch. I've attached an updated version of 
it in this email, for convenience. (It's the same as what I attached to [1].)

The issue you mentioned, of this "unique id", is *already* a problem in the 
currently-released version (in Debian testing and unstable) because of the 
GCC+dpkg-buildpackage changes. So I think it doesn't make much sense to keep 
holding this patch off now.

The concrete benefit of applying this patch, is that tests.r-b.org would be 
able to produce sensible diffoscope output for glibc, so we could track both 
[1] and any future cases of unreproducibility. Currently it times out for 
glibc, because there are too many differences, due to the patch not being 
applied.

What do you think?

Ximin

[1] Except for #847596, but it still reproduces most of the time.

-- 
GPG: ed25519/56034877E1F87C35
GPG: rsa4096/1318EFAC5FBBDBCE
https://github.com/infinity0/pubkeys.git
diff -Nru glibc-2.24/debian/changelog glibc-2.24/debian/changelog
--- glibc-2.24/debian/changelog 2016-11-25 22:09:39.0 +
+++ glibc-2.24/debian/changelog 2016-11-15 18:03:37.0 +
@@ -1,3 +1,10 @@
+glibc (2.24-7.1) UNRELEASED; urgency=medium
+
+  * Non-maintainer upload.
+  * Try for reproducibility.
+
+ -- Ximin Luo   Tue, 15 Nov 2016 
18:03:37 +
+
 glibc (2.24-7) unstable; urgency=medium
 
   [ Samuel Thibault ]
diff -Nru glibc-2.24/debian/rules glibc-2.24/debian/rules
--- glibc-2.24/debian/rules 2016-11-25 21:59:04.0 +
+++ glibc-2.24/debian/rules 2016-11-15 18:03:37.0 +
@@ -45,6 +45,7 @@
 
 DEB_SOURCE_PACKAGE := $(strip $(shell egrep '^Source: ' debian/control | cut 
-f 2 -d ':'))
 
+SOURCE_DATE_EPOCH ?= $(shell dpkg-parsechangelog -STimestamp)
 DEB_VERSION := $(shell dpkg-parsechangelog | egrep '^Version:' | cut -f 2 -d ' 
')
 GLIBC_VERSION = $(shell echo $(DEB_VERSION) | sed -e 's/.*://' -e 's/[+-].*//')
 
diff -Nru glibc-2.24/debian/rules.d/build.mk glibc-2.24/debian/rules.d/build.mk
--- glibc-2.24/debian/rules.d/build.mk  2016-11-25 12:02:24.0 +
+++ glibc-2.24/debian/rules.d/build.mk  2016-11-15 18:03:37.0 +
@@ -316,18 +316,16 @@
 $(stamp)source: $(stamp)patch
mkdir -p $(build-tree)
cd .. && \
-  find $(GLIBC_SOURCES) -depth -newermt '$(DEB_BUILD_DATE)' \
-   -print0 | \
-   xargs -0r touch --no-dereference --date='$(DEB_BUILD_DATE)'
-   cd .. && \
find $(GLIBC_SOURCES) -print0 | \
LC_ALL=C sort -z | \
tar -c -J --null -T - --no-recursion \
--mode=go=rX,u+rw,a-s \
+   --clamp-mtime --mtime "@$(SOURCE_DATE_EPOCH)" \
--owner=root --group=root --numeric-owner \
-f $(CURDIR)/$(build-tree)/glibc-$(GLIBC_VERSION).tar.xz
mkdir -p debian/glibc-source/usr/src/glibc
tar cf - --files-from debian/glibc-source.filelist \
+   --clamp-mtime --mtime "@$(SOURCE_DATE_EPOCH)" \
  | tar -x -C debian/glibc-source/usr/src/glibc -f -
 
touch $@
diff -Nru glibc-2.24/debian/rules.d/debhelper.mk 
glibc-2.24/debian/rules.d/debhelper.mk
--- glibc-2.24/debian/rules.d/debhelper.mk  2016-11-25 22:08:30.0 
+
+++ glibc-2.24/debian/rules.d/debhelper.mk  2016-11-15 18:03:37.0 
+
@@ -77,8 +77,7 @@
-exec chmod a+x '{}' ';'
dh_makeshlibs -Xgconv/ -p$(curpass) -V "$(call xx,shlib_dep)"
# Add relevant udeb: lines in shlibs files
-   chmod a+x debian/shlibs-add-udebs
-   ./debian/shlibs-add-udebs $(curpass)
+   sh ./debian/shlibs-add-udebs $(curpass)
 
dh_installdeb -p$(curpass)
dh_shlibdeps -p$(curpass)

Bug#783210: [Reproducible-builds] Bug#783210: glibc: please make the package build reproducibly

[Ximin Luo]

Ximin Luo:
> Aurelien Jarno:
>> On 2016-06-06 02:48, Ximin Luo wrote:
>>> On Mon, 19 Oct 2015 10:11:25 +0200 Aurelien Jarno  
>>> wrote:
 It means that the point 3 (usage of __DATE__ and __TIME__) is the only
 one left to address in the version currently in experimental.

>>>
>>> Hi, GCC have now added support for SOURCE_DATE_EPOCH so we can use this 
>>> instead. Debhelper now sets this environment variable automatically, and 
>>> GCC should automatically read it, so perhaps we can get rid of the whole 
>>> (3)-related patch, but I haven't tested this yet.
>>
>> In the current code __DATE__ and __TIME__ are used to generate a unique
>> serial number shared between two processes. Therefore using
>> SOURCE_DATE_EPOCH for that is not correct here.
>>
>> I have submitted a patch upstream to fix that another way [1].
>> Unfortunately there is still work to do and I am currently lacking time
>> to work on that. Help would be appreciated.
>>
>> Aurelien
>>
>> [1] https://sourceware.org/ml/libc-alpha/2016-03/msg00240.html
>>
> 
> Thanks Aurelien,
> 
> I reviewed your patch and the discussion linked. I wonder if a quick solution 
> to the issues raise by Mike could be solved by also hashing in the contents 
> of config.status? AIUI autotools, this would include both the have-selinux 
> stuff as well as the alignment values of the build.
> 
> Mike, what do you think? If good, I will adapt Aurelien's patch accordingly 
> and resubmit to the mailing list.
> 

On second thoughts, this wouldn't actually make the build reproducible since 
config.status contains things like SHELL which we want to vary whilst retaining 
reproducibility.

An alternative approach: why not just use the constants defined in version.h, 
plus the size of the statdata struct itself? (The latter would account for 
have-selinux, alignment sizes, as well as any future additions.) Something like 
this:


static const char compilation[] = RELEASE "-" VERSION "-";
[..]
struct statdata
{
  char version[sizeof (compilation) + sizeof (size_t)];
  [..]
}
size_t statdata_size = sizeof (struct statdata);

// later
send_stats (
[..]
memcpy(data.version, compilation, sizeof(compilation));
memcpy(data.version + sizeof(compilation), _size, sizeof(size_t));
// alternatively, we could use itoa if you want human-readable
*(data.version + sizeof(data.version) - 1) = 0;
[..]


My C is not the best so this could probably be tidied up quite a lot, but at 
this stage let's just talk about the general idea first.

X

-- 
GPG: ed25519/56034877E1F87C35
GPG: rsa4096/1318EFAC5FBBDBCE
https://github.com/infinity0/pubkeys.git

Bug#783210: glibc: please make the package build reproducibly

[Ximin Luo]

X-Debbugs-Cc: reproducible-bui...@lists.alioth.debian.org

+CC Mike Frysinger

Aurelien Jarno:
> On 2016-06-06 02:48, Ximin Luo wrote:
>> On Mon, 19 Oct 2015 10:11:25 +0200 Aurelien Jarno  
>> wrote:
>>> It means that the point 3 (usage of __DATE__ and __TIME__) is the only
>>> one left to address in the version currently in experimental.
>>>
>>
>> Hi, GCC have now added support for SOURCE_DATE_EPOCH so we can use this 
>> instead. Debhelper now sets this environment variable automatically, and GCC 
>> should automatically read it, so perhaps we can get rid of the whole 
>> (3)-related patch, but I haven't tested this yet.
> 
> In the current code __DATE__ and __TIME__ are used to generate a unique
> serial number shared between two processes. Therefore using
> SOURCE_DATE_EPOCH for that is not correct here.
> 
> I have submitted a patch upstream to fix that another way [1].
> Unfortunately there is still work to do and I am currently lacking time
> to work on that. Help would be appreciated.
> 
> Aurelien
> 
> [1] https://sourceware.org/ml/libc-alpha/2016-03/msg00240.html
> 

Thanks Aurelien,

I reviewed your patch and the discussion linked. I wonder if a quick solution 
to the issues raise by Mike could be solved by also hashing in the contents of 
config.status? AIUI autotools, this would include both the have-selinux stuff 
as well as the alignment values of the build.

Mike, what do you think? If good, I will adapt Aurelien's patch accordingly and 
resubmit to the mailing list.

X

-- 
GPG: ed25519/56034877E1F87C35
GPG: rsa4096/1318EFAC5FBBDBCE
https://github.com/infinity0/pubkeys.git

Bug#783210: glibc: please make the package build reproducibly

[Aurelien Jarno]

On 2016-06-06 02:48, Ximin Luo wrote:
> On Mon, 19 Oct 2015 10:11:25 +0200 Aurelien Jarno  
> wrote:
> > It means that the point 3 (usage of __DATE__ and __TIME__) is the only
> > one left to address in the version currently in experimental.
> > 
> 
> Hi, GCC have now added support for SOURCE_DATE_EPOCH so we can use this 
> instead. Debhelper now sets this environment variable automatically, and GCC 
> should automatically read it, so perhaps we can get rid of the whole 
> (3)-related patch, but I haven't tested this yet.

In the current code __DATE__ and __TIME__ are used to generate a unique
serial number shared between two processes. Therefore using
SOURCE_DATE_EPOCH for that is not correct here.

I have submitted a patch upstream to fix that another way [1].
Unfortunately there is still work to do and I am currently lacking time
to work on that. Help would be appreciated.

Aurelien


[1] https://sourceware.org/ml/libc-alpha/2016-03/msg00240.html

-- 
Aurelien Jarno  GPG: 4096R/1DDD8C9B
aurel...@aurel32.net http://www.aurel32.net


signature.asc
Description: PGP signature

Bug#783210: glibc: please make the package build reproducibly

[Ximin Luo]

On Mon, 19 Oct 2015 10:11:25 +0200 Aurelien Jarno  wrote:
> It means that the point 3 (usage of __DATE__ and __TIME__) is the only
> one left to address in the version currently in experimental.
> 

Hi, GCC have now added support for SOURCE_DATE_EPOCH so we can use this 
instead. Debhelper now sets this environment variable automatically, and GCC 
should automatically read it, so perhaps we can get rid of the whole 
(3)-related patch, but I haven't tested this yet.

X

-- 
GPG: ed25519/56034877E1F87C35
GPG: rsa4096/1318EFAC5FBBDBCE
git://github.com/infinity0/pubkeys.git

Bug#783210: glibc: please make the package build reproducibly

[Aurelien Jarno]

On 2015-08-23 20:13, Jérémy Bobbio wrote:
> Aurelien Jarno:
> > I have just applied the part concerning point 1. For the 2 other points,
> > from what I have understood there are now patches for gcc to define
> > __DATE__ and __TIME__. So the question is should we still want to get
> > this changes in the glibc? In that case I would try to get these patches
> > upstream.
> 
> It's still unclear if GCC upstream will accept support for setting
> predefined values to __DATE__ and __TIME__. As we are currently lacking
> feedback on the patch that was submitted, waiting would make sense.

Note that point 2 has actually been fixed upstream in the following
commit:

| commit 5006148ee57c80a787b275c04587678e9fe0a3f3
| Author: Roland McGrath 
| Date:   Fri Mar 14 16:05:10 2014 -0700
| 
| Remove "Compiled on ..." crapola from version text.

It means that the point 3 (usage of __DATE__ and __TIME__) is the only
one left to address in the version currently in experimental.

-- 
Aurelien Jarno  GPG: 4096R/1DDD8C9B
aurel...@aurel32.net http://www.aurel32.net


signature.asc
Description: Digital signature

Bug#783210: glibc: please make the package build reproducibly

[Jérémy Bobbio]

Aurelien Jarno:
 I have just applied the part concerning point 1. For the 2 other points,
 from what I have understood there are now patches for gcc to define
 __DATE__ and __TIME__. So the question is should we still want to get
 this changes in the glibc? In that case I would try to get these patches
 upstream.

It's still unclear if GCC upstream will accept support for setting
predefined values to __DATE__ and __TIME__. As we are currently lacking
feedback on the patch that was submitted, waiting would make sense.

-- 
Lunar.''`. 
lu...@debian.org: :Ⓐ  :  # apt-get install anarchism
`. `'` 
  `-   


signature.asc
Description: Digital signature

Bug#783210: glibc: please make the package build reproducibly

[Aurelien Jarno]

On 2015-04-23 23:30, Jérémy Bobbio wrote:
 Source: glibc
 Version: 2.19-18
 Severity: wishlist
 Tags: patch
 User: reproducible-bui...@lists.alioth.debian.org
 Usertags: timestamps fileordering umask username uname
 
 Hi!
 
 While working on the “reproducible builds” effort [1], we have noticed
 that glibc could not be built reproducibly.
 
 The attached patch addresses several issues:
 
 1. The source tarball will now stay identical despite variations of
the time of the build, user, group, umask and file ordering.
 2. version-info.h currently captures the build time and the current
kernel version. In the context of Debian this is not really useful
and a new patch simply removes it. The behavior is now the same
if built under Linux or not.
 3. nscd uses the date and time of the build as a version marker. So a
patch is added to allow the build date to be set externally. The date
of the latest debian/changelog entry will be used instead of the
current time for Debian.
 
 Once applied, glibc can be built reproducibly in our current
 experimental framework.

I have just applied the part concerning point 1. For the 2 other points,
from what I have understood there are now patches for gcc to define
__DATE__ and __TIME__. So the question is should we still want to get
this changes in the glibc? In that case I would try to get these patches
upstream.

Aurelien

-- 
Aurelien Jarno  GPG: 4096R/1DDD8C9B
aurel...@aurel32.net http://www.aurel32.net


signature.asc
Description: Digital signature

Bug#783210: glibc: please make the package build reproducibly

[Jérémy Bobbio]

Source: glibc
Version: 2.19-18
Severity: wishlist
Tags: patch
User: reproducible-bui...@lists.alioth.debian.org
Usertags: timestamps fileordering umask username uname

Hi!

While working on the “reproducible builds” effort [1], we have noticed
that glibc could not be built reproducibly.

The attached patch addresses several issues:

1. The source tarball will now stay identical despite variations of
   the time of the build, user, group, umask and file ordering.
2. version-info.h currently captures the build time and the current
   kernel version. In the context of Debian this is not really useful
   and a new patch simply removes it. The behavior is now the same
   if built under Linux or not.
3. nscd uses the date and time of the build as a version marker. So a
   patch is added to allow the build date to be set externally. The date
   of the latest debian/changelog entry will be used instead of the
   current time for Debian.

Once applied, glibc can be built reproducibly in our current
experimental framework.

 [1]: https://wiki.debian.org/ReproducibleBuilds

-- 
Lunar.''`. 
lu...@debian.org: :Ⓐ  :  # apt-get install anarchism
`. `'` 
  `-   
diff --git a/debian/changelog b/debian/changelog
index a06fc11..fbb6d32 100644
--- a/debian/changelog
+++ b/debian/changelog
@@ -1,3 +1,18 @@
+glibc (2.19-18.0~reproducible1) UNRELEASED; urgency=low
+
+  * Make the package build reproducibly:
+- any/local-reproducible-version-info.diff: new patch to remove build
+  system information and build date from version-info.h.
+- any/unsubmitted-allow-to-set-build-date.diff: allow to set the
+  build date through an environment variable when running ./configure.
+  This is then used in nscd.
+- Set the previously mentioned build date to the latest debian/changelog
+  entry.
+- Create source tarball in a deterministic manner: adjust file
+  modification time, user, group, permissions, and file order.
+
+ -- Jérémy Bobbio lu...@debian.org  Thu, 23 Apr 2015 11:18:59 +0200
+
 glibc (2.19-18) unstable; urgency=medium
 
   [ Aurelien Jarno ]
diff --git a/debian/patches/any/local-reproducible-version-info.diff b/debian/patches/any/local-reproducible-version-info.diff
new file mode 100644
index 000..06012ea
--- /dev/null
+++ b/debian/patches/any/local-reproducible-version-info.diff
@@ -0,0 +1,49 @@
+To enable glibc to build reproducibly, we remove build system information
+and build date from version-info.h.
+
+diff --git a/csu/Makefile b/csu/Makefile
+index b5afea0..211c2e6 100644
+--- a/csu/Makefile
 b/csu/Makefile
+@@ -133,40 +133,7 @@ all-Banner-files = $(wildcard $(addsuffix /Banner,\
+ 		 $(sysdeps-srcdirs
+ $(objpfx)version-info.h: $(common-objpfx)config.make $(all-Banner-files)
+ 	$(make-target-directory)
+-	(case $(config-os) in \
+-	   linux*) version=`(printf '%s\n%s\n' \
+-'#include linux/version.h' \
+-UTS_RELEASE \
+-			 | $(CC) $(CPPFLAGS) -O -E -P - -DNOT_IN_libc=1 | \
+-			 sed -e 's/\([^]*\).*/\1/p' -e d) 2/dev/null`;\
+-		   if [ -z $$version ]; then \
+-		 version=`(printf '%s\n%s\n' \
+-  '#include linux/version.h' \
+-   LINUX_VERSION_CODE \
+-			   | $(CC) $(CPPFLAGS) -O -E -P - -DNOT_IN_libc=1 \
+-			   | sed -n -e '/^[123456789].*/p' \
+-			   | awk '{v=$$1; \
+-   printf(%d.%d.%d\n, \
+-	  v/65535, v/256%256, v%256)}') \
+-2/dev/null`; \
+-		   fi; \
+-		   if [ -z $$version ]; then \
+-		 if [ -r /proc/version ]; then \
+-		   version=`sed 's/.*Linux version \([^ ]*\) .*/\1/' \
+- /proc/version`; \
+-		 else \
+-		   version=`uname -r`; \
+-		 fi; \
+-		   fi; \
+-		   os=`uname -s 2 /dev/null`; \
+-		   if [ -z $$os ]; then \
+-		 os=Linux; \
+-		   fi; \
+-		   printf 'Compiled on a %s %s system on %s.\\n\n' \
+-			  $$os $$version `date +%Y-%m-%d`;; \
+-	   *) ;; \
+-	 esac; \
+-	 files=$(all-Banner-files);\
++	(files=$(all-Banner-files);\
+ 	 if test -n $$files; then\
+ 	   printf 'Available extensions:\\n\n';		\
+ 	   sed -e '/^#/d' -e 's/^[[:space:]]*/	/'		\
diff --git a/debian/patches/any/unsubmitted-allow-to-set-build-date.diff b/debian/patches/any/unsubmitted-allow-to-set-build-date.diff
new file mode 100644
index 000..df7662a
--- /dev/null
+++ b/debian/patches/any/unsubmitted-allow-to-set-build-date.diff
@@ -0,0 +1,75 @@
+nscd uses the date and time of the build as a version marker. In order to allow
+builds to be reproducible, we now allow the date to be set by the environment
+variable BUILD_DATE when running ./configure.
+
+diff --git a/config.h.in b/config.h.in
+index 40797e7..02ccb2d 100644
+--- a/config.h.in
 b/config.h.in
+@@ -246,3 +246,6 @@
+ #undef HAVE_MIPS_NAN2008
+ 
+ #endif
++
++/* Date and time of the build. */
++#undef BUILD_DATE
+diff --git a/configure.ac b/configure.ac
+index 

Bug#783210: glibc: please make the package build reproducibly

[Jérémy Bobbio]

Jérémy Bobbio:
 3. nscd uses the date and time of the build as a version marker. So a
patch is added to allow the build date to be set externally. The date
of the latest debian/changelog entry will be used instead of the
current time for Debian.

Reiner Herrmann noticed that the patch would produce different result
depending on the locale. Updated patch attached.

-- 
Lunar.''`. 
lu...@debian.org: :Ⓐ  :  # apt-get install anarchism
`. `'` 
  `-   
diff --git a/debian/changelog b/debian/changelog
index a06fc11..fbb6d32 100644
--- a/debian/changelog
+++ b/debian/changelog
@@ -1,3 +1,18 @@
+glibc (2.19-18.0~reproducible1) UNRELEASED; urgency=low
+
+  * Make the package build reproducibly:
+- any/local-reproducible-version-info.diff: new patch to remove build
+  system information and build date from version-info.h.
+- any/unsubmitted-allow-to-set-build-date.diff: allow to set the
+  build date through an environment variable when running ./configure.
+  This is then used in nscd.
+- Set the previously mentioned build date to the latest debian/changelog
+  entry.
+- Create source tarball in a deterministic manner: adjust file
+  modification time, user, group, permissions, and file order.
+
+ -- Jérémy Bobbio lu...@debian.org  Thu, 23 Apr 2015 11:18:59 +0200
+
 glibc (2.19-18) unstable; urgency=medium
 
   [ Aurelien Jarno ]
diff --git a/debian/patches/any/local-reproducible-version-info.diff b/debian/patches/any/local-reproducible-version-info.diff
new file mode 100644
index 000..06012ea
--- /dev/null
+++ b/debian/patches/any/local-reproducible-version-info.diff
@@ -0,0 +1,49 @@
+To enable glibc to build reproducibly, we remove build system information
+and build date from version-info.h.
+
+diff --git a/csu/Makefile b/csu/Makefile
+index b5afea0..211c2e6 100644
+--- a/csu/Makefile
 b/csu/Makefile
+@@ -133,40 +133,7 @@ all-Banner-files = $(wildcard $(addsuffix /Banner,\
+ 		 $(sysdeps-srcdirs
+ $(objpfx)version-info.h: $(common-objpfx)config.make $(all-Banner-files)
+ 	$(make-target-directory)
+-	(case $(config-os) in \
+-	   linux*) version=`(printf '%s\n%s\n' \
+-'#include linux/version.h' \
+-UTS_RELEASE \
+-			 | $(CC) $(CPPFLAGS) -O -E -P - -DNOT_IN_libc=1 | \
+-			 sed -e 's/\([^]*\).*/\1/p' -e d) 2/dev/null`;\
+-		   if [ -z $$version ]; then \
+-		 version=`(printf '%s\n%s\n' \
+-  '#include linux/version.h' \
+-   LINUX_VERSION_CODE \
+-			   | $(CC) $(CPPFLAGS) -O -E -P - -DNOT_IN_libc=1 \
+-			   | sed -n -e '/^[123456789].*/p' \
+-			   | awk '{v=$$1; \
+-   printf(%d.%d.%d\n, \
+-	  v/65535, v/256%256, v%256)}') \
+-2/dev/null`; \
+-		   fi; \
+-		   if [ -z $$version ]; then \
+-		 if [ -r /proc/version ]; then \
+-		   version=`sed 's/.*Linux version \([^ ]*\) .*/\1/' \
+- /proc/version`; \
+-		 else \
+-		   version=`uname -r`; \
+-		 fi; \
+-		   fi; \
+-		   os=`uname -s 2 /dev/null`; \
+-		   if [ -z $$os ]; then \
+-		 os=Linux; \
+-		   fi; \
+-		   printf 'Compiled on a %s %s system on %s.\\n\n' \
+-			  $$os $$version `date +%Y-%m-%d`;; \
+-	   *) ;; \
+-	 esac; \
+-	 files=$(all-Banner-files);\
++	(files=$(all-Banner-files);\
+ 	 if test -n $$files; then\
+ 	   printf 'Available extensions:\\n\n';		\
+ 	   sed -e '/^#/d' -e 's/^[[:space:]]*/	/'		\
diff --git a/debian/patches/any/unsubmitted-allow-to-set-build-date.diff b/debian/patches/any/unsubmitted-allow-to-set-build-date.diff
new file mode 100644
index 000..df7662a
--- /dev/null
+++ b/debian/patches/any/unsubmitted-allow-to-set-build-date.diff
@@ -0,0 +1,75 @@
+nscd uses the date and time of the build as a version marker. In order to allow
+builds to be reproducible, we now allow the date to be set by the environment
+variable BUILD_DATE when running ./configure.
+
+diff --git a/config.h.in b/config.h.in
+index 40797e7..02ccb2d 100644
+--- a/config.h.in
 b/config.h.in
+@@ -246,3 +246,6 @@
+ #undef HAVE_MIPS_NAN2008
+ 
+ #endif
++
++/* Date and time of the build. */
++#undef BUILD_DATE
+diff --git a/configure.ac b/configure.ac
+index f3dd87d..aa0b30d 100644
+--- a/configure.ac
 b/configure.ac
+@@ -2173,6 +2173,11 @@ RELEASE=`sed -n -e 's/^#define RELEASE \([^]*\)/\1/p'  $srcdir/version.h`
+ AC_SUBST(VERSION)
+ AC_SUBST(RELEASE)
+ 
++if x$BUILD_DATE = x; then
++	BUILD_DATE=`LC_ALL=C date -u '+%b %_d %Y %H:%M:%S'`
++fi
++AC_DEFINE_UNQUOTED([BUILD_DATE], [$BUILD_DATE], Date and time of the build.)
++
+ AC_CONFIG_FILES([config.make Makefile])
+ AC_CONFIG_COMMANDS([default],[[
+ case $CONFIG_FILES in *config.make*)
+diff --git a/configure b/configure
+index fc023d0..b03be3f 100755
+--- a/configure
 b/configure
+@@ -7387,6 +7394,15 @@
+ 
+ 
+ 
++if x$BUILD_DATE = x; then
++	BUILD_DATE=`LC_ALL=C date -u '+%b %_d %Y