Bug#783210: glibc: please make the package build reproducibly

2016-06-05 Thread Ximin Luo 
On Mon, 19 Oct 2015 10:11:25 +0200 Aurelien Jarno  wrote:
> It means that the point 3 (usage of __DATE__ and __TIME__) is the only
> one left to address in the version currently in experimental.
> 

Hi, GCC have now added support for SOURCE_DATE_EPOCH so we can use this 
instead. Debhelper now sets this environment variable automatically, and GCC 
should automatically read it, so perhaps we can get rid of the whole 
(3)-related patch, but I haven't tested this yet.

X

-- 
GPG: ed25519/56034877E1F87C35
GPG: rsa4096/1318EFAC5FBBDBCE
git://github.com/infinity0/pubkeys.git

Bug#783210: [Reproducible-builds] Bug#783210: glibc: please make the package build reproducibly

2016-07-22 Thread Ximin Luo 
Ximin Luo:
> Aurelien Jarno:
>> On 2016-06-06 02:48, Ximin Luo wrote:
>>> On Mon, 19 Oct 2015 10:11:25 +0200 Aurelien Jarno <aurel...@aurel32.net> 
>>> wrote:
>>>> It means that the point 3 (usage of __DATE__ and __TIME__) is the only
>>>> one left to address in the version currently in experimental.
>>>>
>>>
>>> Hi, GCC have now added support for SOURCE_DATE_EPOCH so we can use this 
>>> instead. Debhelper now sets this environment variable automatically, and 
>>> GCC should automatically read it, so perhaps we can get rid of the whole 
>>> (3)-related patch, but I haven't tested this yet.
>>
>> In the current code __DATE__ and __TIME__ are used to generate a unique
>> serial number shared between two processes. Therefore using
>> SOURCE_DATE_EPOCH for that is not correct here.
>>
>> I have submitted a patch upstream to fix that another way [1].
>> Unfortunately there is still work to do and I am currently lacking time
>> to work on that. Help would be appreciated.
>>
>> Aurelien
>>
>> [1] https://sourceware.org/ml/libc-alpha/2016-03/msg00240.html
>>
> 
> Thanks Aurelien,
> 
> I reviewed your patch and the discussion linked. I wonder if a quick solution 
> to the issues raise by Mike could be solved by also hashing in the contents 
> of config.status? AIUI autotools, this would include both the have-selinux 
> stuff as well as the alignment values of the build.
> 
> Mike, what do you think? If good, I will adapt Aurelien's patch accordingly 
> and resubmit to the mailing list.
> 

On second thoughts, this wouldn't actually make the build reproducible since 
config.status contains things like SHELL which we want to vary whilst retaining 
reproducibility.

An alternative approach: why not just use the constants defined in version.h, 
plus the size of the statdata struct itself? (The latter would account for 
have-selinux, alignment sizes, as well as any future additions.) Something like 
this:


static const char compilation[] = RELEASE "-" VERSION "-";
[..]
struct statdata
{
  char version[sizeof (compilation) + sizeof (size_t)];
  [..]
}
size_t statdata_size = sizeof (struct statdata);

// later
send_stats (
[..]
memcpy(data.version, compilation, sizeof(compilation));
memcpy(data.version + sizeof(compilation), _size, sizeof(size_t));
// alternatively, we could use itoa if you want human-readable
*(data.version + sizeof(data.version) - 1) = 0;
[..]


My C is not the best so this could probably be tidied up quite a lot, but at 
this stage let's just talk about the general idea first.

X

-- 
GPG: ed25519/56034877E1F87C35
GPG: rsa4096/1318EFAC5FBBDBCE
https://github.com/infinity0/pubkeys.git

Bug#783210: glibc: please make the package build reproducibly

2016-07-22 Thread Ximin Luo 
X-Debbugs-Cc: reproducible-bui...@lists.alioth.debian.org

+CC Mike Frysinger

Aurelien Jarno:
> On 2016-06-06 02:48, Ximin Luo wrote:
>> On Mon, 19 Oct 2015 10:11:25 +0200 Aurelien Jarno <aurel...@aurel32.net> 
>> wrote:
>>> It means that the point 3 (usage of __DATE__ and __TIME__) is the only
>>> one left to address in the version currently in experimental.
>>>
>>
>> Hi, GCC have now added support for SOURCE_DATE_EPOCH so we can use this 
>> instead. Debhelper now sets this environment variable automatically, and GCC 
>> should automatically read it, so perhaps we can get rid of the whole 
>> (3)-related patch, but I haven't tested this yet.
> 
> In the current code __DATE__ and __TIME__ are used to generate a unique
> serial number shared between two processes. Therefore using
> SOURCE_DATE_EPOCH for that is not correct here.
> 
> I have submitted a patch upstream to fix that another way [1].
> Unfortunately there is still work to do and I am currently lacking time
> to work on that. Help would be appreciated.
> 
> Aurelien
> 
> [1] https://sourceware.org/ml/libc-alpha/2016-03/msg00240.html
> 

Thanks Aurelien,

I reviewed your patch and the discussion linked. I wonder if a quick solution 
to the issues raise by Mike could be solved by also hashing in the contents of 
config.status? AIUI autotools, this would include both the have-selinux stuff 
as well as the alignment values of the build.

Mike, what do you think? If good, I will adapt Aurelien's patch accordingly and 
resubmit to the mailing list.

X

-- 
GPG: ed25519/56034877E1F87C35
GPG: rsa4096/1318EFAC5FBBDBCE
https://github.com/infinity0/pubkeys.git

Bug#844420: FTBFS: tests fail on hosts without network access

2016-11-15 Thread Ximin Luo 
Package: glibc
Version: 2.24-5
Severity: important
Tags: upstream patch
Forwarded: https://sourceware.org/bugzilla/show_bug.cgi?id=20826

Dear Maintainer,

posix/tst-getaddrinfo.c is causing glibc to FTBFS on 
tests.reproducible-builds.org:

https://tests.reproducible-builds.org/debian/rbuild/unstable/amd64/glibc_2.24-5.rbuild.log.gz

The attached patch should fix this; I gave a more detailed description in the 
upstream bug report.

It would be good if this were already applied to the Debian glibc so that we 
can begin to see diffoscope output again and work on its reproducibility.

Thanks,
X

-- System Information:
Debian Release: stretch/sid
  APT prefers testing
  APT policy: (990, 'testing'), (500, 'stable'), (300, 'unstable'), (200, 
'experimental')
Architecture: amd64 (x86_64)
Foreign Architectures: i386

Kernel: Linux 4.7.0-1-amd64 (SMP w/4 CPU cores)
Locale: LANG=en_GB.utf8, LC_CTYPE=en_GB.utf8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)
Description: Don't condition failure of double-dot-FQDN on single-dot-FQDN
 Otherwise the test fails on systems that disable network access during build
 time, since getaddrinfo() fails in all cases and the test returns 2.
 .
 Normally, such systems can still resolve "localhost." via nsswitch/getent and
 getaddrinfo is not suppose to attempt resolution of "localhost." anyways.
Author: Ximin Luo <infini...@debian.org>
Bug: TBD
---
This patch header follows DEP-3: http://dep.debian.net/deps/dep3/
--- a/posix/tst-getaddrinfo5.c
+++ b/posix/tst-getaddrinfo5.c
@@ -36,20 +36,6 @@
   size_t len = strlen (host);
   struct addrinfo *ai;
 
-  /* If the name doesn't resolve with a single dot at the
-	 end, skip it.  */
-  host[len-1] = 0;
-  if (getaddrinfo (host, NULL, NULL, ) != 0)
-	{
-	  printf ("resolving \"%s\" failed, skipping this hostname\n", host);
-	  continue;
-	}
-  printf ("resolving \"%s\" worked, proceeding to test\n", host);
-  freeaddrinfo (ai);
-
-  /* If it resolved with a single dot, check that it doesn't with
-	 a second trailing dot.  */
-  host[len-1] = '.';
   if (getaddrinfo (host, NULL, NULL, ) == 0)
 	{
 	  printf ("resolving \"%s\" worked, test failed\n", host);

Bug#844420: FTBFS: tests fail on hosts without network access

2016-11-16 Thread Ximin Luo 
Aurelien Jarno:
> On 2016-11-15 16:00, Ximin Luo wrote:
>> Package: glibc
>> Version: 2.24-5
>> Severity: important
>> Tags: upstream patch
>> Forwarded: https://sourceware.org/bugzilla/show_bug.cgi?id=20826
>>
>> Dear Maintainer,
>>
>> posix/tst-getaddrinfo.c is causing glibc to FTBFS on 
>> tests.reproducible-builds.org:
>>
>> https://tests.reproducible-builds.org/debian/rbuild/unstable/amd64/glibc_2.24-5.rbuild.log.gz
>>
>> The attached patch should fix this; I gave a more detailed description in 
>> the upstream bug report.
> 
> Hum, I am not sure it is the correct way to fix that. I think that
> libnss_files should be able to resolve entries from /etc/hosts when the
> query is relative, but also when it is fully qualified. This is how
> libnss_dns behaves.
> 

Looks like I was wrong before about getaddrinfo bypassing /etc/hosts, and it 
does indeed look at /etc/hosts.

$ sudo -u sbuild getent hosts localhost
127.0.0.1   profitbricks-build17-amd64.debian.net 
profitbricks-build17-amd64 localhost
$ sudo -u sbuild getent hosts localhost.
# no results

However if you change "localhost" in /etc/hosts to "localhost." then the above 
results would be reversed.

What do you think the full behaviour should be?

> Also note that technically the glibc doesn't require network access,
> just a DNS server able to resolve 'localhost.'
> 

So how do you want to fix this? glibc doesn't currently build-depend on a name 
server, and I assume you wouldn't want to do that. Can you give me some hints 
on what to patch, to fix libnss_files?

X

-- 
GPG: ed25519/56034877E1F87C35
GPG: rsa4096/1318EFAC5FBBDBCE
https://github.com/infinity0/pubkeys.git

Bug#844420: FTBFS: tests fail on hosts without network access

2016-11-17 Thread Ximin Luo 
Aurelien Jarno:
> On 2016-11-16 09:48, Ximin Luo wrote:
>> Aurelien Jarno:
>>> On 2016-11-15 16:00, Ximin Luo wrote:
>>>> Package: glibc
>>>> Version: 2.24-5
>>>> Severity: important
>>>> Tags: upstream patch
>>>> Forwarded: https://sourceware.org/bugzilla/show_bug.cgi?id=20826
>>>>
>>>> Dear Maintainer,
>>>>
>>>> posix/tst-getaddrinfo.c is causing glibc to FTBFS on 
>>>> tests.reproducible-builds.org:
>>>>
>>>> https://tests.reproducible-builds.org/debian/rbuild/unstable/amd64/glibc_2.24-5.rbuild.log.gz
>>>>
>>>> The attached patch should fix this; I gave a more detailed description in 
>>>> the upstream bug report.
>>>
>>> Hum, I am not sure it is the correct way to fix that. I think that
>>> libnss_files should be able to resolve entries from /etc/hosts when the
>>> query is relative, but also when it is fully qualified. This is how
>>> libnss_dns behaves.
>>>
>>
>> Looks like I was wrong before about getaddrinfo bypassing /etc/hosts, and it 
>> does indeed look at /etc/hosts.
>>
>> $ sudo -u sbuild getent hosts localhost
>> 127.0.0.1   profitbricks-build17-amd64.debian.net 
>> profitbricks-build17-amd64 localhost
>> $ sudo -u sbuild getent hosts localhost.
>> # no results
>>
>> However if you change "localhost" in /etc/hosts to "localhost." then the 
>> above results would be reversed.
>>
>> What do you think the full behaviour should be?
> 
> I think that it should returns 127.0.0.1 for both 'localhost' and
> 'localhost.'
> 

It is more complex than just making "localhost." work. As mentioned, if one 
explicitly adds "localhost." into /etc/hosts then it already works.

So the question is whether libnss_files should attempt to interpret relative 
names in /etc/hosts as if they were absolute names. That is what you are 
effectively proposing. (I assume that you're not proposing everyone to add 
extra duplicate entries into their /etc/hosts file.)

I don't know which sorts of standards documents describe this file or whatever, 
but I guess it would be quite a major change. Anyway I will go ask upstream.

>>> Also note that technically the glibc doesn't require network access,
>>> just a DNS server able to resolve 'localhost.'
>>>
>>
>> So how do you want to fix this? glibc doesn't currently build-depend on a 
>> name server, and I assume you wouldn't want to do that. Can you give me some 
>> hints on what to patch, to fix libnss_files?
> 
> My point about that, is that a package should build without network
> access. Now I am not sure it actually means without even a nameserver.
> 
> Anyway I don't know that part of the code a lot, but the fix has
> probably to be done in nss/nss_files/files-hosts.c.
> 
> Aurelien
> 

-- 
GPG: ed25519/56034877E1F87C35
GPG: rsa4096/1318EFAC5FBBDBCE
https://github.com/infinity0/pubkeys.git

Bug#783210: [PATCH] nscd_stat.c: make the build reproducible

2016-11-04 Thread Ximin Luo 
Mike Frysinger:
> On 28 Jul 2016 15:15, Florian Weimer wrote:
>> On 03/09/2016 05:30 PM, Mike Frysinger wrote:
>>> would it be so terrible to properly marshall this data ?
>>
>> Ximin Luo and I discussed this and I wonder if it is possible to read 
>> out the libc.so.6 build ID if it is present.  It should indirectly call 
>> all the layout dependencies and be reasonably easy to access because it 
>> is in an allocated section (and we might want to print it from an 
>> libc.so.6 invocation, too).
>>
>> We still need the time-based approach if the build ID is not available, 
>> but I expect most distributions will have something like it.
>>
>> The Debian bug is:
>>
>>https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=783210
>>
>> (Also Cc:ed)
> 
> agreed that build-id should be an acceptable replacement for what the
> code is doing today, but in order to pull that off, i guess you'd have
> to have to do a configure test to see if build-id is active ?  if you
> leave the logic to runtime, you'd still need to include the datetime
> stamp in the object which would still make the build unreproducible.
> 
> this also doesn't really cover the quoted idea of marshalling the data
> between client & server :).
> -mike
> 

Hi all,

I've written a small program that prints out the Build IDs of all the objects 
that are dynamically linked to it, plus itself.

It works well, although I'm not a C expert so I don't know if it is portable 
enough. For example, I hard-code some >>2 <<2s in there, along with a uint8_t - 
I didn't see a corresponding ElfW(xxx) type in elf.h

Another downside is it needs to be linked against libdl, which I think is not 
the case currently with nscd. I'm not sure if this carries extra security risk 
or whatever.

An alternative would be to detect the build-id *at build time* and then 
monkey-patch it into the binary itself.

What do you all think? How shall I proceed?

X

-- 
GPG: ed25519/56034877E1F87C35
GPG: rsa4096/1318EFAC5FBBDBCE
https://github.com/infinity0/pubkeys.git
#define _GNU_SOURCE
#include 
#include 

int callback (struct dl_phdr_info *info, size_t size, void *data) {
  printf ("\nname: %s\n", info->dlpi_name);

  ElfW(Phdr) *phdr = (ElfW(Phdr) *) info->dlpi_phdr;
  for (ElfW(Half) i = 0; i < info->dlpi_phnum; i++) {
if (phdr->p_type == PT_NOTE) {
  ElfW(Addr) addr = info->dlpi_addr + info->dlpi_phdr[i].p_vaddr;
  ElfW(Addr) nend = addr + info->dlpi_phdr[i].p_memsz;
  //printf ("found NOTE segment at: %p to %p\n", addr, nend);

  while (addr < nend) {
	ElfW(Nhdr) *nhdr = (ElfW(Nhdr) *) addr;
	// According to the ELF spec, namesz and descsz do not include padding
	// but that's how they're laid out in memory; add the padding here.
	ElfW(Addr) nameoff = (((nhdr->n_namesz-1)>>2)+1)<<2;
	ElfW(Addr) descoff = (((nhdr->n_descsz-1)>>2)+1)<<2;

	if (nhdr->n_type == NT_GNU_BUILD_ID) {
	  const uint8_t *buf = (const uint8_t *) ((ElfW(Addr))(nhdr + 1) + nameoff);
	  printf("Build ID");
	  for (int j = 0; j < nhdr->n_descsz; j++)
	printf(":%02X", buf[j]);
	  printf("\n");
	}

	//printf("skipping section type %02X\n", nhdr->n_type);
	addr = (ElfW(Addr))(nhdr + 1) + nameoff + descoff;
  }
}

phdr += 1;
  }

  return 0;
}

int main() {
  dl_iterate_phdr(callback, NULL);
}

Bug#783210: [PATCH] nscd_stat.c: make the build reproducible

2016-11-04 Thread Ximin Luo 
Ximin Luo:
> Mike Frysinger:
>> On 28 Jul 2016 15:15, Florian Weimer wrote:
>>> On 03/09/2016 05:30 PM, Mike Frysinger wrote:
>>>> would it be so terrible to properly marshall this data ?
>>>
>>> Ximin Luo and I discussed this and I wonder if it is possible to read 
>>> out the libc.so.6 build ID if it is present.  It should indirectly call 
>>> all the layout dependencies and be reasonably easy to access because it 
>>> is in an allocated section (and we might want to print it from an 
>>> libc.so.6 invocation, too).
>>>
>>> We still need the time-based approach if the build ID is not available, 
>>> but I expect most distributions will have something like it.
>>>
>>> The Debian bug is:
>>>
>>>https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=783210
>>>
>>> (Also Cc:ed)
>>
>> agreed that build-id should be an acceptable replacement for what the
>> code is doing today, but in order to pull that off, i guess you'd have
>> to have to do a configure test to see if build-id is active ?  if you
>> leave the logic to runtime, you'd still need to include the datetime
>> stamp in the object which would still make the build unreproducible.
>>
>> this also doesn't really cover the quoted idea of marshalling the data
>> between client & server :).
>> -mike
>>
> 
> Hi all,
> 
> I've written a small program that prints out the Build IDs of all the objects 
> that are dynamically linked to it, plus itself.
> 
> It works well, although I'm not a C expert so I don't know if it is portable 
> enough. For example, I hard-code some >>2 <<2s in there, along with a uint8_t 
> - I didn't see a corresponding ElfW(xxx) type in elf.h
> 
> Another downside is it needs to be linked against libdl, which I think is not 
> the case currently with nscd. I'm not sure if this carries extra security 
> risk or whatever.
> 

Oh! Actually it doesn't need to be linked against libdl. That was from an 
earlier version of the code where I was using dlinfo instead of 
dl_iterate_phdr. But this latter function doesn't need extra libs. :)

> An alternative would be to detect the build-id *at build time* and then 
> monkey-patch it into the binary itself.
> 
> What do you all think? How shall I proceed?
> 

X

-- 
GPG: ed25519/56034877E1F87C35
GPG: rsa4096/1318EFAC5FBBDBCE
https://github.com/infinity0/pubkeys.git

Bug#847478: FTBFS: needs to Build-Depends on gperf

2016-12-08 Thread Ximin Luo 
Package: glibc
Version: 2.24-7
Severity: serious
Tags: patch
Justification: fails to build from source (but built successfully in the past)

Dear Maintainer,

whilst trying to build glibc:

/usr/bin/install -c -m 644 
/tmp/debrepro.uGH5xEsmL1/build/source/build-tree/amd64-libc/gnu/lib-names-64.h 
/tmp/debrepro.uGH5xEsmL1/build/source/debian/tmp-libc/usr/include/gnu/lib-names-64.h
/usr/bin/install -c -m 644 
/tmp/debrepro.uGH5xEsmL1/build/source/build-tree/amd64-libc/libBrokenLocale.map 
/tmp/debrepro.uGH5xEsmL1/build/source/debian/tmp-libc/usr/lib/x86_64-linux-gnu/libBrokenLocale_pic.map
cd programs \
&& gperf -acCgopt -k1,2,5,9,$ -L ANSI-C -N charmap_hash charmap-kw.gperf > 
charmap-kw.h.new
/bin/bash: line 1: gperf: command not found
Makefile:66: recipe for target 'programs/charmap-kw.h' failed
make[3]: *** [programs/charmap-kw.h] Error 127
make[3]: Leaving directory '/tmp/debrepro.uGH5xEsmL1/build/source/locale'
Makefile:230: recipe for target 'locale/subdir_install' failed
make[2]: *** [locale/subdir_install] Error 2
make[2]: Leaving directory '/tmp/debrepro.uGH5xEsmL1/build/source'
Makefile:12: recipe for target 'install' failed
make[1]: *** [install] Error 2
make[1]: Leaving directory 
'/tmp/debrepro.uGH5xEsmL1/build/source/build-tree/amd64-libc'
debian/rules.d/build.mk:162: recipe for target 
'/tmp/debrepro.uGH5xEsmL1/build/source/stamp-dir/install_libc' failed
make: *** [/tmp/debrepro.uGH5xEsmL1/build/source/stamp-dir/install_libc] Error 2
dpkg-buildpackage: error: fakeroot debian/rules binary gave error exit status 2

I: artifacts left in /tmp/debrepro.uGH5xEsmL1

Probably this just needs a Build-Depends: gperf, I am trying the build again 
now.

X

-- System Information:
Debian Release: stretch/sid
  APT prefers testing
  APT policy: (990, 'testing'), (500, 'stable'), (300, 'unstable'), (200, 
'experimental')
Architecture: amd64 (x86_64)
Foreign Architectures: i386

Kernel: Linux 4.7.0-1-amd64 (SMP w/4 CPU cores)
Locale: LANG=en_GB.utf8, LC_CTYPE=en_GB.utf8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)

Bug#783210: glibc: please make the package build reproducibly

2016-12-09 Thread Ximin Luo 
- the upstream folks

Aurelien Jarno:
> On 2016-06-06 02:48, Ximin Luo wrote:
>> On Mon, 19 Oct 2015 10:11:25 +0200 Aurelien Jarno <aurel...@aurel32.net> 
>> wrote:
>>> It means that the point 3 (usage of __DATE__ and __TIME__) is the only
>>> one left to address in the version currently in experimental.
>>>
>>
>> Hi, GCC have now added support for SOURCE_DATE_EPOCH so we can use this 
>> instead. Debhelper now sets this environment variable automatically, and GCC 
>> should automatically read it, so perhaps we can get rid of the whole 
>> (3)-related patch, but I haven't tested this yet.
> 
> In the current code __DATE__ and __TIME__ are used to generate a unique
> serial number shared between two processes. Therefore using
> SOURCE_DATE_EPOCH for that is not correct here.
> 
> I have submitted a patch upstream to fix that another way [1].
> Unfortunately there is still work to do and I am currently lacking time
> to work on that. Help would be appreciated.
> 

Hi Aurelian, I noticed that due to GCC supporting SOURCE_DATE_EPOCH and 
dpkg-buildpackage automatically setting this now, glibc is actually already 
reproducible [1] if you apply this patch. I've attached an updated version of 
it in this email, for convenience. (It's the same as what I attached to [1].)

The issue you mentioned, of this "unique id", is *already* a problem in the 
currently-released version (in Debian testing and unstable) because of the 
GCC+dpkg-buildpackage changes. So I think it doesn't make much sense to keep 
holding this patch off now.

The concrete benefit of applying this patch, is that tests.r-b.org would be 
able to produce sensible diffoscope output for glibc, so we could track both 
[1] and any future cases of unreproducibility. Currently it times out for 
glibc, because there are too many differences, due to the patch not being 
applied.

What do you think?

Ximin

[1] Except for #847596, but it still reproduces most of the time.

-- 
GPG: ed25519/56034877E1F87C35
GPG: rsa4096/1318EFAC5FBBDBCE
https://github.com/infinity0/pubkeys.git
diff -Nru glibc-2.24/debian/changelog glibc-2.24/debian/changelog
--- glibc-2.24/debian/changelog 2016-11-25 22:09:39.0 +
+++ glibc-2.24/debian/changelog 2016-11-15 18:03:37.0 +
@@ -1,3 +1,10 @@
+glibc (2.24-7.1) UNRELEASED; urgency=medium
+
+  * Non-maintainer upload.
+  * Try for reproducibility.
+
+ -- Ximin Luo <infinity0@profitbricks-build17-amd64>  Tue, 15 Nov 2016 
18:03:37 +
+
 glibc (2.24-7) unstable; urgency=medium
 
   [ Samuel Thibault ]
diff -Nru glibc-2.24/debian/rules glibc-2.24/debian/rules
--- glibc-2.24/debian/rules 2016-11-25 21:59:04.0 +
+++ glibc-2.24/debian/rules 2016-11-15 18:03:37.0 +
@@ -45,6 +45,7 @@
 
 DEB_SOURCE_PACKAGE := $(strip $(shell egrep '^Source: ' debian/control | cut 
-f 2 -d ':'))
 
+SOURCE_DATE_EPOCH ?= $(shell dpkg-parsechangelog -STimestamp)
 DEB_VERSION := $(shell dpkg-parsechangelog | egrep '^Version:' | cut -f 2 -d ' 
')
 GLIBC_VERSION = $(shell echo $(DEB_VERSION) | sed -e 's/.*://' -e 's/[+-].*//')
 
diff -Nru glibc-2.24/debian/rules.d/build.mk glibc-2.24/debian/rules.d/build.mk
--- glibc-2.24/debian/rules.d/build.mk  2016-11-25 12:02:24.0 +
+++ glibc-2.24/debian/rules.d/build.mk  2016-11-15 18:03:37.0 +
@@ -316,18 +316,16 @@
 $(stamp)source: $(stamp)patch
mkdir -p $(build-tree)
cd .. && \
-  find $(GLIBC_SOURCES) -depth -newermt '$(DEB_BUILD_DATE)' \
-   -print0 | \
-   xargs -0r touch --no-dereference --date='$(DEB_BUILD_DATE)'
-   cd .. && \
find $(GLIBC_SOURCES) -print0 | \
LC_ALL=C sort -z | \
tar -c -J --null -T - --no-recursion \
--mode=go=rX,u+rw,a-s \
+   --clamp-mtime --mtime "@$(SOURCE_DATE_EPOCH)" \
--owner=root --group=root --numeric-owner \
-f $(CURDIR)/$(build-tree)/glibc-$(GLIBC_VERSION).tar.xz
mkdir -p debian/glibc-source/usr/src/glibc
tar cf - --files-from debian/glibc-source.filelist \
+   --clamp-mtime --mtime "@$(SOURCE_DATE_EPOCH)" \
  | tar -x -C debian/glibc-source/usr/src/glibc -f -
 
touch $@
diff -Nru glibc-2.24/debian/rules.d/debhelper.mk 
glibc-2.24/debian/rules.d/debhelper.mk
--- glibc-2.24/debian/rules.d/debhelper.mk  2016-11-25 22:08:30.0 
+
+++ glibc-2.24/debian/rules.d/debhelper.mk  2016-11-15 18:03:37.0 
+
@@ -77,8 +77,7 @@
-exec chmod a+x '{}' ';'
dh_makeshlibs -Xgconv/ -p$(curpass) -V "$(call xx,shlib_dep)"
# Add relevant udeb: lines in shlibs files
-   chmod a+x debian/shlibs-add-udebs
-   ./debian/shlibs-add-udebs $(curpass)
+   sh ./debian/shlibs-add-udebs $(curpass)
 
dh_installdeb -p$(curpass)
dh_shlibdeps -p$(curpass)

Bug#847596: locales-all non-deterministically generates either kk_KZ.PT154 or kk_KZ.RK1048 despite declaring support for both

2016-12-09 Thread Ximin Luo 
Package: locales-all
Version: 2.24-7
Severity: normal

Dear Maintainer,

The build of locales-all sometimes generates kk_KZ.PT154 (about 1/6 of the
time) and sometimes generates kk_KZ.RK1048 (about 5/6 of the time), even though
it declares support for both. Note also that RK1048 is patched in, via
debian/patches/localedata/supported.diff. You can find my test files here:

https://people.debian.org/~infinity0/res/debrepro/

Some of them are attached, for your convenience.

To build these, I applied glibc.debdiff to 2.24-7 then ran `debrepro` on the
unpacked source tree. Note they are reproducible except for locales-all. Note
also that `debrepro` does not reliably reproduce this unreproducibility, you
have to run it a few times to see it. I ran it 3 times, for a total of 6
builds, before I saw this. Then:

$ sudo dpkg -i first/libc6_2.24-7.1_amd64.deb
[..]
$ sudo dpkg -i first/libc-bin_2.24-7.1_amd64.deb
[..]
$ sudo dpkg -i first/libc-l10n_2.24-7.1_all.deb
[..]
$ sudo dpkg -i first/locales_2.24-7.1_all.deb
[..]

$ sudo dpkg -i first/locales-all_2.24-7.1_amd64.deb
[..]
$ python3 ./localetest.py 
** error trying to set locale to kk_KZ.PT154
locale: Cannot set LC_CTYPE to default locale: No such file or directory
locale: Cannot set LC_MESSAGES to default locale: No such file or directory
locale: Cannot set LC_ALL to default locale: No such file or directory

$ sudo dpkg -i second/locales-all_2.24-7.1_amd64.deb 
[..]
$ python3 ./localetest.py
** error trying to set locale to kk_KZ.RK1048
locale: Cannot set LC_CTYPE to default locale: No such file or directory
locale: Cannot set LC_MESSAGES to default locale: No such file or directory
locale: Cannot set LC_ALL to default locale: No such file or directory


-- System Information:
Debian Release: stretch/sid
  APT prefers testing
  APT policy: (990, 'testing'), (500, 'unstable-debug'), (500, 
'testing-debug'), (300, 'unstable'), (200, 'experimental'), (1, 
'experimental-debug')
Architecture: amd64 (x86_64)
Foreign Architectures: i386

Kernel: Linux 4.6.0-1-amd64 (SMP w/4 CPU cores)
Locale: LANG=en_GB.utf8, LC_CTYPE=en_GB.utf8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)
diff -Nru glibc-2.24/debian/changelog glibc-2.24/debian/changelog
--- glibc-2.24/debian/changelog	2016-11-25 22:09:39.0 +
+++ glibc-2.24/debian/changelog	2016-11-15 18:03:37.0 +
@@ -1,3 +1,10 @@
+glibc (2.24-7.1) UNRELEASED; urgency=medium
+
+  * Non-maintainer upload.
+  * Try for reproducibility.
+
+ -- Ximin Luo <infinity0@profitbricks-build17-amd64>  Tue, 15 Nov 2016 18:03:37 +
+
 glibc (2.24-7) unstable; urgency=medium
 
   [ Samuel Thibault ]
diff -Nru glibc-2.24/debian/rules glibc-2.24/debian/rules
--- glibc-2.24/debian/rules	2016-11-25 21:59:04.0 +
+++ glibc-2.24/debian/rules	2016-11-15 18:03:37.0 +
@@ -45,6 +45,7 @@
 
 DEB_SOURCE_PACKAGE := $(strip $(shell egrep '^Source: ' debian/control | cut -f 2 -d ':'))
 
+SOURCE_DATE_EPOCH ?= $(shell dpkg-parsechangelog -STimestamp)
 DEB_VERSION := $(shell dpkg-parsechangelog | egrep '^Version:' | cut -f 2 -d ' ')
 GLIBC_VERSION = $(shell echo $(DEB_VERSION) | sed -e 's/.*://' -e 's/[+-].*//')
 
diff -Nru glibc-2.24/debian/rules.d/build.mk glibc-2.24/debian/rules.d/build.mk
--- glibc-2.24/debian/rules.d/build.mk	2016-11-25 12:02:24.0 +
+++ glibc-2.24/debian/rules.d/build.mk	2016-11-15 18:03:37.0 +
@@ -316,18 +316,16 @@
 $(stamp)source: $(stamp)patch
 	mkdir -p $(build-tree)
 	cd .. && \
-	   find $(GLIBC_SOURCES) -depth -newermt '$(DEB_BUILD_DATE)' \
-			-print0 | \
-   xargs -0r touch --no-dereference --date='$(DEB_BUILD_DATE)'
-	cd .. && \
 		find $(GLIBC_SOURCES) -print0 | \
 		LC_ALL=C sort -z | \
 		tar -c -J --null -T - --no-recursion \
 			--mode=go=rX,u+rw,a-s \
+			--clamp-mtime --mtime "@$(SOURCE_DATE_EPOCH)" \
 			--owner=root --group=root --numeric-owner \
 			-f $(CURDIR)/$(build-tree)/glibc-$(GLIBC_VERSION).tar.xz
 	mkdir -p debian/glibc-source/usr/src/glibc
 	tar cf - --files-from debian/glibc-source.filelist \
+		--clamp-mtime --mtime "@$(SOURCE_DATE_EPOCH)" \
 	  | tar -x -C debian/glibc-source/usr/src/glibc -f -
 
 	touch $@
diff -Nru glibc-2.24/debian/rules.d/debhelper.mk glibc-2.24/debian/rules.d/debhelper.mk
--- glibc-2.24/debian/rules.d/debhelper.mk	2016-11-25 22:08:30.0 +
+++ glibc-2.24/debian/rules.d/debhelper.mk	2016-11-15 18:03:37.0 +
@@ -77,8 +77,7 @@
 		-exec chmod a+x '{}' ';'
 	dh_makeshlibs -Xgconv/ -p$(curpass) -V "$(call xx,shlib_dep)"
 	# Add relevant udeb: lines in shlibs files
-	chmod a+x debian/shlibs-add-udebs
-	./debian/shlibs-add-udebs $(curpass)
+	sh ./debian/shlibs-add-udebs $(curpass)
 
 	dh_installdeb -p$(curpass)
 	dh_shlibdeps -p$(curpass)
#!/usr/bin/python3

import subprocess
import sys
import os

with open("/usr/share/i18n/SUPPORTED") as fp:
for line in fp.readlines():

Bug#783210: glibc: please make the package build reproducibly

2016-12-19 Thread Ximin Luo 
Aurelien Jarno:
> [..]
> 
> It would be nice to have an explanation why the changes from your patch
> are needed. See my comments below.
> 
> 
>> diff -Nru glibc-2.24/debian/rules glibc-2.24/debian/rules
>> --- glibc-2.24/debian/rules  2016-11-25 21:59:04.0 +
>> +++ glibc-2.24/debian/rules  2016-11-15 18:03:37.0 +
>> @@ -45,6 +45,7 @@
>>  
>>  DEB_SOURCE_PACKAGE := $(strip $(shell egrep '^Source: ' debian/control | 
>> cut -f 2 -d ':'))
>>  
>> +SOURCE_DATE_EPOCH ?= $(shell dpkg-parsechangelog -STimestamp)
>>  DEB_VERSION := $(shell dpkg-parsechangelog | egrep '^Version:' | cut -f 2 
>> -d ' ')
>>  GLIBC_VERSION = $(shell echo $(DEB_VERSION) | sed -e 's/.*://' -e 
>> 's/[+-].*//')
> 
> You told above that SOURCE_DATE_EPOCH is already set by
> dpkg-buildpackage, so why do we need to do that?
> 

It is not strictly necessary, but it allows one to build reproducibly even when 
not using dpkg-buildpackage, e.g. when trying to re-run a specific part of the 
build via `debian/rules build` or something else. This is helpful to save time 
when debugging.

>> diff -Nru glibc-2.24/debian/rules.d/build.mk 
>> glibc-2.24/debian/rules.d/build.mk
>> --- glibc-2.24/debian/rules.d/build.mk   2016-11-25 12:02:24.0 
>> +
>> +++ glibc-2.24/debian/rules.d/build.mk   2016-11-15 18:03:37.0 
>> +
>> @@ -316,18 +316,16 @@
>>  $(stamp)source: $(stamp)patch
>>  mkdir -p $(build-tree)
>>  cd .. && \
>> -   find $(GLIBC_SOURCES) -depth -newermt '$(DEB_BUILD_DATE)' \
>> --print0 | \
>> -   xargs -0r touch --no-dereference --date='$(DEB_BUILD_DATE)'
>> -cd .. && \
>>  find $(GLIBC_SOURCES) -print0 | \
>>  LC_ALL=C sort -z | \
>>  tar -c -J --null -T - --no-recursion \
>>  --mode=go=rX,u+rw,a-s \
>> +--clamp-mtime --mtime "@$(SOURCE_DATE_EPOCH)" \
>>  --owner=root --group=root --numeric-owner \
>>  -f $(CURDIR)/$(build-tree)/glibc-$(GLIBC_VERSION).tar.xz
>>  mkdir -p debian/glibc-source/usr/src/glibc
>>  tar cf - --files-from debian/glibc-source.filelist \
>> +--clamp-mtime --mtime "@$(SOURCE_DATE_EPOCH)" \
>>| tar -x -C debian/glibc-source/usr/src/glibc -f -
> 
> Why do we need to change this, what doesn't work in the current code? The 
> current code
> has been provided by Jérémy Bobbio in the current bug.
> 

Yes, it doesn't work. but also it was not applied correctly:

1. The original patch had a mistake in: we need "@$(unix_timestamp)", the '@' 
tells GNU find/tar/date/etc to interpret the subsequent value as a UNIX 
timestamp. Otherwise these tools will complain about an invalid date, but 
silently use a different value.

2. The original patch was not applied fully/correctly either - nowhere in the 
current glibc packaging is DEB_BUILD_DATE actually set. The original patch set 
this, instead of my previous hunk that sets SOURCE_DATE_EPOCH, however this is 
not present in the current glibc packaging.

3. Since the original patch was written, we got the --clamp-mtime feature into 
GNU tar, and this allows us to make the patch smaller and the resulting code is 
neater. (It also avoids writing directly to the filesystem.)

>>  touch $@
>> diff -Nru glibc-2.24/debian/rules.d/debhelper.mk 
>> glibc-2.24/debian/rules.d/debhelper.mk
>> --- glibc-2.24/debian/rules.d/debhelper.mk   2016-11-25 22:08:30.0 
>> +
>> +++ glibc-2.24/debian/rules.d/debhelper.mk   2016-11-15 18:03:37.0 
>> +
>> @@ -77,8 +77,7 @@
>>  -exec chmod a+x '{}' ';'
>>  dh_makeshlibs -Xgconv/ -p$(curpass) -V "$(call xx,shlib_dep)"
>>  # Add relevant udeb: lines in shlibs files
>> -chmod a+x debian/shlibs-add-udebs
>> -./debian/shlibs-add-udebs $(curpass)
>> +sh ./debian/shlibs-add-udebs $(curpass)
>>  
>>  dh_installdeb -p$(curpass)
>>  dh_shlibdeps -p$(curpass)
> 
> Why do we need this change?
> 

This is not strictly necessary, but it helps one perform reproductions in the 
same source tree, by running dpkg-buildpackage twice. Otherwise, glibc-source 
has -x for that file in the first build and +x for that file in the second 
build.

Yes, our testing infrastructure does not run builds twice in the same source 
tree and neither does debrepro nor reprotest, however doing this is useful for 
debugging purposes, to save time and avoid having to re-run the whole build 
againg.

X

-- 
GPG: ed25519/56034877E1F87C35
GPG: rsa4096/1318EFAC5FBBDBCE
https://github.com/infinity0/pubkeys.git