Bug#864082: fontconfig: please make the cache files reproducible

2022-01-11 Thread Johannes Schauer Marin Rodrigues 
Hi,

Quoting Johannes Schauer Marin Rodrigues (2021-12-28 09:38:51)
> since this bug has not seen maintainer action for several months now and
> because fontconfig is one of the last remaining bits that make a
> Priority:Standard chroot unreproducible, I'd like to propose to NMU
> fontconfig with the attached patch.
> 
> Essentially, I took the patch by Chris Lamb and modified it so that
> uuid_generate_sha1() will be used if SOURCE_DATE_EPOCH is set. The original
> behaviour is retained without SOURCE_DATE_EPOCH being set. This should be the
> solution with zero impact on normal installations while making those
> installations that care (like reproducible chroot or image creation)
> reproducible.
> 
> If I don't get a NACK from you on this, I plan to upload to DELAYED/10 in two
> weeks.

I just uploaded fontcontig to DELAYED/10 with the attached debdiff.

Thanks!

cheers, joschdiff -Nru fontconfig-2.13.1/debian/changelog fontconfig-2.13.1/debian/changelog
--- fontconfig-2.13.1/debian/changelog	2020-05-15 12:55:02.0 +0200
+++ fontconfig-2.13.1/debian/changelog	2022-01-12 07:49:42.0 +0100
@@ -1,3 +1,10 @@
+fontconfig (2.13.1-4.3) unstable; urgency=medium
+
+  * Non-maintainer upload.
+  * Make cache files reproducible if SOURCE_DATE_EPOCH is set (closes: #864082)
+
+ -- Johannes Schauer Marin Rodrigues   Wed, 12 Jan 2022 07:49:42 +0100
+
 fontconfig (2.13.1-4.2) unstable; urgency=medium
 
   * Non-maintainer upload.
diff -Nru fontconfig-2.13.1/debian/patches/0001-Make-the-cache-filenames-determinstic.patch fontconfig-2.13.1/debian/patches/0001-Make-the-cache-filenames-determinstic.patch
--- fontconfig-2.13.1/debian/patches/0001-Make-the-cache-filenames-determinstic.patch	1970-01-01 01:00:00.0 +0100
+++ fontconfig-2.13.1/debian/patches/0001-Make-the-cache-filenames-determinstic.patch	2022-01-12 07:49:31.0 +0100
@@ -0,0 +1,49 @@
+From 70565e0f73d116a2a9523146228efcca1e76b016 Mon Sep 17 00:00:00 2001
+From: Chris Lamb 
+Date: Mon, 29 Oct 2018 15:48:51 -0400
+Subject: [PATCH] Make the cache filenames determinstic
+
+Whilst working on the Reproducible Builds[0] effort, we noticed that
+fontconfig generates cache files with unreproducible/non-deterministic
+filenames.
+
+This is a supplement to the changes added in f098adac54ab where we
+ensured that the checksums themselves were determistic but the files
+that were stored in the cache directory are currently being given
+"random" names via uuid(3)'s uuid_generate_random function, thus
+any images that generate such files have different contents on every
+build.
+
+This patch changes the behaviour of the cache directory filename
+calculation to be based on the "source" directory name, rather than
+being entirely random.
+
+An alternative solution could be to continue to use the previous
+uuid_generate_random function but use this alternative codepath if the
+SOURCE_DATE_EPOCH[1] environment variable was determined to be
+present via getenv(3).
+
+This work was sponsored by Tails[2] and tracked in Debian in #864082[3].
+
+ [0] https://reproducible-builds.org/
+ [1] https://reproducible-builds.org/specs/source-date-epoch/
+ [2] https://tails.boum.org/
+ [3] https://bugs.debian.org/864082
+---
+ src/fccache.c | 2 +-
+ 1 file changed, 1 insertion(+), 1 deletion(-)
+
+--- a/src/fccache.c
 b/src/fccache.c
+@@ -101,7 +101,10 @@ FcDirCacheCreateUUID (FcChar8  *dir,
+ 	ret = FcFalse;
+ 	goto bail3;
+ 	}
+-	uuid_generate_random (uuid);
++	if (getenv("SOURCE_DATE_EPOCH"))
++	uuid_generate_sha1 (uuid, target, target, strlen(target));
++	else
++	uuid_generate_random (uuid);
+ 	if (force)
+ 	hash_add = FcHashTableReplace;
+ 	else
diff -Nru fontconfig-2.13.1/debian/patches/series fontconfig-2.13.1/debian/patches/series
--- fontconfig-2.13.1/debian/patches/series	2020-05-13 12:18:40.0 +0200
+++ fontconfig-2.13.1/debian/patches/series	2022-01-12 07:49:31.0 +0100
@@ -6,3 +6,4 @@
 0001-fix-memory-leaks.patch
 0002-more-memory-leaks.patch
 0003-Set-name_mapping-to-NULL-after-freeing.patch
+0001-Make-the-cache-filenames-determinstic.patch
diff -Nru fontconfig-2.13.1/debian/po/pt.po fontconfig-2.13.1/debian/po/pt.po
--- fontconfig-2.13.1/debian/po/pt.po	2020-05-13 15:16:46.0 +0200
+++ fontconfig-2.13.1/debian/po/pt.po	2022-01-12 07:49:42.0 +0100
@@ -87,21 +87,18 @@
 #. Type: select
 #. Description
 #: ../fontconfig-config.templates:3002
-#| msgid "Font tuning method for screen (system default):"
 msgid "Automatic font hinting style:"
 msgstr "Estilo de desenho do tipo de letra automático:"
 
 #. Type: select
 #. Description
 #: ../fontconfig-config.templates:3002
-#| msgid ""
-#| "Please select the preferred method for tuning fonts for screen rendering."
 msgid ""
 "Please select the preferred automatic font hinting style to be used as the "
 "system default."
 msgstr ""
-"Por favor seleccione o estilo de desenho do tipo de letra automático a "
-"ser usado como predefinição do sistema."
+"Por

Bug#864082: fontconfig: please make the cache files reproducible

2021-12-28 Thread Johannes Schauer Marin Rodrigues 
Hi,

since this bug has not seen maintainer action for several months now and
because fontconfig is one of the last remaining bits that make a
Priority:Standard chroot unreproducible, I'd like to propose to NMU fontconfig
with the attached patch.

Essentially, I took the patch by Chris Lamb and modified it so that
uuid_generate_sha1() will be used if SOURCE_DATE_EPOCH is set. The original
behaviour is retained without SOURCE_DATE_EPOCH being set. This should be the
solution with zero impact on normal installations while making those
installations that care (like reproducible chroot or image creation)
reproducible.

If I don't get a NACK from you on this, I plan to upload to DELAYED/10 in two
weeks.

Thanks!

cheers, joschdiff -Nru fontconfig-2.13.1/debian/changelog fontconfig-2.13.1/debian/changelog
--- fontconfig-2.13.1/debian/changelog	2020-05-15 12:55:02.0 +0200
+++ fontconfig-2.13.1/debian/changelog	2021-12-27 20:53:49.0 +0100
@@ -1,3 +1,10 @@
+fontconfig (2.13.1-4.3) UNRELEASED; urgency=medium
+
+  * Non-maintainer upload.
+  * Make cache files reproducible (closes: #864082)
+
+ -- Johannes Schauer Marin Rodrigues   Mon, 27 Dec 2021 20:53:49 +0100
+
 fontconfig (2.13.1-4.2) unstable; urgency=medium
 
   * Non-maintainer upload.
diff -Nru fontconfig-2.13.1/debian/patches/0001-Make-the-cache-filenames-determinstic.patch fontconfig-2.13.1/debian/patches/0001-Make-the-cache-filenames-determinstic.patch
--- fontconfig-2.13.1/debian/patches/0001-Make-the-cache-filenames-determinstic.patch	1970-01-01 01:00:00.0 +0100
+++ fontconfig-2.13.1/debian/patches/0001-Make-the-cache-filenames-determinstic.patch	2021-12-27 20:53:49.0 +0100
@@ -0,0 +1,49 @@
+From 70565e0f73d116a2a9523146228efcca1e76b016 Mon Sep 17 00:00:00 2001
+From: Chris Lamb 
+Date: Mon, 29 Oct 2018 15:48:51 -0400
+Subject: [PATCH] Make the cache filenames determinstic
+
+Whilst working on the Reproducible Builds[0] effort, we noticed that
+fontconfig generates cache files with unreproducible/non-deterministic
+filenames.
+
+This is a supplement to the changes added in f098adac54ab where we
+ensured that the checksums themselves were determistic but the files
+that were stored in the cache directory are currently being given
+"random" names via uuid(3)'s uuid_generate_random function, thus
+any images that generate such files have different contents on every
+build.
+
+This patch changes the behaviour of the cache directory filename
+calculation to be based on the "source" directory name, rather than
+being entirely random.
+
+An alternative solution could be to continue to use the previous
+uuid_generate_random function but use this alternative codepath if the
+SOURCE_DATE_EPOCH[1] environment variable was determined to be
+present via getenv(3).
+
+This work was sponsored by Tails[2] and tracked in Debian in #864082[3].
+
+ [0] https://reproducible-builds.org/
+ [1] https://reproducible-builds.org/specs/source-date-epoch/
+ [2] https://tails.boum.org/
+ [3] https://bugs.debian.org/864082
+---
+ src/fccache.c | 2 +-
+ 1 file changed, 1 insertion(+), 1 deletion(-)
+
+--- a/src/fccache.c
 b/src/fccache.c
+@@ -101,7 +101,10 @@ FcDirCacheCreateUUID (FcChar8  *dir,
+ 	ret = FcFalse;
+ 	goto bail3;
+ 	}
+-	uuid_generate_random (uuid);
++	if (getenv("SOURCE_DATE_EPOCH"))
++	uuid_generate_sha1 (uuid, target, target, strlen(target));
++	else
++	uuid_generate_random (uuid);
+ 	if (force)
+ 	hash_add = FcHashTableReplace;
+ 	else
diff -Nru fontconfig-2.13.1/debian/patches/series fontconfig-2.13.1/debian/patches/series
--- fontconfig-2.13.1/debian/patches/series	2020-05-13 12:18:40.0 +0200
+++ fontconfig-2.13.1/debian/patches/series	2021-12-27 20:53:43.0 +0100
@@ -6,3 +6,4 @@
 0001-fix-memory-leaks.patch
 0002-more-memory-leaks.patch
 0003-Set-name_mapping-to-NULL-after-freeing.patch
+0001-Make-the-cache-filenames-determinstic.patch


signature.asc
Description: signature

Bug#864082: fontconfig: please make the cache files reproducible

2021-11-02 Thread Andres Pavez 
Hello,
I would like to send another gentle ping about this.

This bug is affecting the deployment upgrade of a current reproducible iso.

Thank you very much,
--
Andrés Pavez

On Wed, Jan 6, 2021 at 2:51 AM Johannes Schauer Marin Rodrigues
 wrote:
>
> Dear maintainers,
>
> On Sun, 13 Sep 2020 08:13:06 - "Chris Lamb"  wrote:
> > Friendly ping on this?
>
> I'd like to send another ping about this.
>
> This bug is affecting my package mmdebstrap so I'd love to see it fixed.
>
> Thanks!
>
> cheers, josch

Bug#864082: fontconfig: please make the cache files reproducible

2021-01-06 Thread Johannes Schauer Marin Rodrigues 
Dear maintainers,

On Sun, 13 Sep 2020 08:13:06 - "Chris Lamb"  wrote:
> Friendly ping on this?

I'd like to send another ping about this.

This bug is affecting my package mmdebstrap so I'd love to see it fixed.

Thanks!

cheers, josch

signature.asc
Description: signature

Bug#864082: fontconfig: please make the cache files reproducible

2020-09-13 Thread Chris Lamb 
Chris Lamb wrote:

> [..]

Friendly ping on this?


Regards,

-- 
  ,''`.
 : :'  : Chris Lamb
 `. `'`  la...@debian.org / chris-lamb.co.uk
   `-

Bug#864082: fontconfig: please make the cache files reproducible

2018-10-29 Thread Chris Lamb 
forwarded 864082 
https://lists.freedesktop.org/archives/fontconfig/2018-October/006374.html
thanks

Chris Lamb wrote:

> On this. Indeed, I can reproduce it […]

Can also confirm that SOURCE_DATE_EPOCH is available in the
postinst and to fc-cache itself;locally added some printf(3)
statements to FcDirChecksum.

> @@ -1,5 +1,5 @@
> +7fd806a4-197a-4989-8a34-2c49019d041b-le64.cache-7
> +95c367ca-9c9b-47d7-9625-c03688da4239-le64.cache-7
>  CACHEDIR.TAG
> -a4fcff53-9cdb-4103-baea-3115d0f9e21e-le64.cache-7
> -afd762ff-b72e-4c3f-98f5-19b3b7cf7f95-le64.cache-7
> -c8c796f6-9945-4521-bb11-2ad6a193bcf2-le64.cache-7
> -ee218622-3364-4921-aaae-6e7d011e7c5e-le64.cache-7
> +cb3a236e-83c1-49d4-92f9-a44aa67ef71c-le64.cache-7
> +dc12f21a-6ea1-4373-b9e4-cfc7bd8165f7-le64.cache-7

These are uuid's generated by uuid(3)'s uuid_generate_random.

I've sent a patch upstream to:

  https://lists.freedesktop.org/archives/fontconfig/2018-October/006374.html


Best wishes,

-- 
  ,''`.
 : :'  : Chris Lamb
 `. `'`  la...@debian.org / chris-lamb.co.uk
   `-

Bug#864082: fontconfig: please make the cache files reproducible

2018-10-29 Thread Chris Lamb 
Hi Johannes,

> Unfortunately, fontconfig still installs unreproducibly. Try this:
> 
>  $ sudo SOURCE_DATE_EPOCH=1540078640 debootstrap --variant=minbase --
> include=fontconfig unstable debian-fontconfig1
> [...]
>  $ sudo SOURCE_DATE_EPOCH=1540078640 debootstrap --variant=minbase --
> include=fontconfig unstable debian-fontconfig2

On this. Indeed, I can reproduce it:

$ diffoscope --markdown=- debian-fontconfig*/var/cache/fontconfig

# Comparing debian-fontconfig1/var/cache/fontconfig & 
debian-fontconfig2/var/cache/fontconfig

## file list

@@ -1,5 +1,5 @@
+7fd806a4-197a-4989-8a34-2c49019d041b-le64.cache-7
+95c367ca-9c9b-47d7-9625-c03688da4239-le64.cache-7
 CACHEDIR.TAG
-a4fcff53-9cdb-4103-baea-3115d0f9e21e-le64.cache-7
-afd762ff-b72e-4c3f-98f5-19b3b7cf7f95-le64.cache-7
-c8c796f6-9945-4521-bb11-2ad6a193bcf2-le64.cache-7
-ee218622-3364-4921-aaae-6e7d011e7c5e-le64.cache-7
+cb3a236e-83c1-49d4-92f9-a44aa67ef71c-le64.cache-7
+dc12f21a-6ea1-4373-b9e4-cfc7bd8165f7-le64.cache-7

## stat {}

@@ -1,8 +1,8 @@
 
   Size: 4096   Blocks: 8  IO Block: 4096   directory
 Links: 2
 Access: (0755/drwxr-xr-x)  Uid: (0/root)   Gid: (0/root)
 
-Modify: 2018-10-29 15:00:44.915312284 +
+Modify: 2018-10-29 15:02:00.475850377 +
 
  Birth: -

## Comparing debian-fontconfig1/var/cache/fontconfig/CACHEDIR.TAG & 
debian-fontconfig2/var/cache/fontconfig/CACHEDIR.TAG

### stat {}

@@ -1,8 +1,8 @@
 
   Size: 200Blocks: 8  IO Block: 4096   regular file
 Links: 1
 Access: (0644/-rw-r--r--)  Uid: (0/root)   Gid: (0/root)
 
-Modify: 2018-10-29 15:00:44.915312284 +
+Modify: 2018-10-29 15:02:00.475850377 +
 
  Birth: -


Best wishes,

-- 
  ,''`.
 : :'  : Chris Lamb
 `. `'`  la...@debian.org / chris-lamb.co.uk
   `-

Bug#864082: fontconfig: please make the cache files reproducible

2018-10-20 Thread Johannes Schauer 
On Wed, 26 Sep 2018 19:35:44 +0100 Chris Lamb  wrote:
> Hi Sven,
> 
> > It does:
> 
> Mea culpa; I was looking at 2.13.0-1, not 2.13.1-1. Thanks!

Unfortunately, fontconfig still installs unreproducibly. Try this:

 $ sudo SOURCE_DATE_EPOCH=1540078640 debootstrap --variant=minbase 
--include=fontconfig unstable debian-fontconfig1
[...]
 $ sudo SOURCE_DATE_EPOCH=1540078640 debootstrap --variant=minbase 
--include=fontconfig unstable debian-fontconfig2
[...]
 $ ls debian-fontconfig*/var/cache/fontconfig
debian-fontconfig1/var/cache/fontconfig:
4fad8281-8542-4657-8757-d5df4b311494-le64.cache-7
7120926d-e4ed-4251-ad8b-5ba4ecf03e2e-le64.cache-7
9e33dc3b-4aaa-4051-907f-695857f3058a-le64.cache-7
CACHEDIR.TAG
db908e3c-2a63-46e7-b5bd-4182bf5118bf-le64.cache-7

debian-fontconfig2/var/cache/fontconfig:
4358383d-0ba9-430d-b068-413004d17d4b-le64.cache-7
445e0a61-5266-4ace-8498-73282459b5d4-le64.cache-7
6b839627-b4cb-4965-b36c-5828cf1fcecb-le64.cache-7
90f39d32-e3d9-4be5-8931-d48aa583062e-le64.cache-7
CACHEDIR.TAG

The same effect happens when installing fontconfig via apt in identical chroots
while having SOURCE_DATE_EPOCH set:

 $ sudo debootstrap --variant=minbase unstable debian-fontconfig1
[...]
 $ sudo cp -a debian-fontconfig1 debian-fontconfig2
 $ sudo SOURCE_DATE_EPOCH=1540078640 chroot debian-fontconfig1 apt-get install 
fontconfig
[...]
 $ sudo SOURCE_DATE_EPOCH=1540078640 chroot debian-fontconfig2 apt-get install 
fontconfig
[...]
 $ ls debian-fontconfig*/var/cache/fontconfig
debian-fontconfig1/var/cache/fontconfig:
52b8b1a5-0c31-4d6b-909f-d2a23007c019-le64.cache-7
76f797dc-9b33-4ce9-b7ef-a9fd4cccf9b4-le64.cache-7
a243f91c-f050-4f66-bd7d-7b21aeb36579-le64.cache-7
a84c7f0e-a95b-4af5-876c-609ef74b476a-le64.cache-7
CACHEDIR.TAG

debian-fontconfig2/var/cache/fontconfig:
41954618-f099-4d12-b610-83140cfb0f75-le64.cache-7
4d918e1a-1f74-404f-a16c-0d9e23e1aef2-le64.cache-7
955f4600-4ca9-43fe-9dc1-907b7ec54758-le64.cache-7
CACHEDIR.TAG
e709e1e4-06c1-4dc6-9b3e-72913b5a3e00-le64.cache-7

And yes, the SOURCE_DATE_EPOCH variable is definitely visible in fontconfig's
postinst script where fc-cache is being run.

So what is still missing?

Thanks!

cheers, josch


signature.asc
Description: signature

Bug#864082: fontconfig: please make the cache files reproducible

2018-09-26 Thread Chris Lamb 
Hi Sven,

> It does:

Mea culpa; I was looking at 2.13.0-1, not 2.13.1-1. Thanks!


Regards,

-- 
  ,''`.
 : :'  : Chris Lamb
 `. `'`  la...@debian.org / chris-lamb.co.uk
   `-

Bug#864082: fontconfig: please make the cache files reproducible

2018-09-26 Thread Sven Joachim 
On 2018-09-26 19:24 +0100, Chris Lamb wrote:

> Hi Sven,
>
>> It seems that 2.13.1-1 was that release, can you confirm that?
>
> Thanks for chiming in.
>
> Given that this version does not contain the string SOURCE_DATE_EPOCH,

It does:

,
| $ grep -n SOURCE_DATE_EPOCH **/*
| doc/fontconfig-user.html:1549:>SOURCE_DATE_EPOCHSOURCE_DATE_EPOCHSOURCE_DATE_EPOCHSOURCE_DATE_EPOCH
| doc/fontconfig-user.sgml:807:is used to ensure fc-cache(1) 
generates files in a deterministic manner in order to support reproducible 
builds. When set to a numeric representation of UNIX timestamp, fontconfig will 
prefer this value over using the modification timestamps of the input files in 
order to identify which cache files require regeneration. If 
SOURCE_DATE_EPOCH is not set (or is newer than the mtime of 
the directory), the existing behaviour is unchanged.
| doc/fontconfig-user.sgml:812:fc-cat(1), fc-cache(1), fc-list(1), fc-match(1), 
fc-query(1), https://reproducible-builds.org/specs/source-date-epoch/;>SOURCE_DATE_EPOCH.
| doc/fontconfig-user.txt:790:   SOURCE_DATE_EPOCH is used to ensure 
fc-cache(1) generates files in a
| doc/fontconfig-user.txt:794:   to identify which cache files require 
regeneration. If SOURCE_DATE_EPOCH
| doc/fontconfig-user.txt:801:   [3]SOURCE_DATE_EPOCH.
| doc/fonts-conf.5:756:\fBSOURCE_DATE_EPOCH\fR
| doc/fonts-conf.5:757:is used to ensure fc-cache(1) generates files in a 
deterministic manner in order to support reproducible builds. When set to a 
numeric representation of UNIX timestamp, fontconfig will prefer this value 
over using the modification timestamps of the input files in order to identify 
which cache files require regeneration. If SOURCE_DATE_EPOCH is not set (or is 
newer than the mtime of the directory), the existing behaviour is unchanged.
| doc/fonts-conf.5:760:fc-cat(1), fc-cache(1), fc-list(1), fc-match(1), 
fc-query(1), SOURCE_DATE_EPOCH 
https://reproducible-builds.org/specs/source-date-epoch/>\&.
| src/fccache.c:1079:source_date_epoch = getenv("SOURCE_DATE_EPOCH");
| src/fccache.c:1086:  "Fontconfig: SOURCE_DATE_EPOCH 
invalid\n");
| src/fccache.c:1090:  "Fontconfig: SOURCE_DATE_EPOCH: 
strtoull: %s: %llu\n",
| src/fccache.c:1094:  "Fontconfig: SOURCE_DATE_EPOCH has 
trailing garbage\n");
| src/fccache.c:1097:  "Fontconfig: SOURCE_DATE_EPOCH must be 
<= %lu but saw: %llu\n",
| src/fccache.c:1112:if (getenv("SOURCE_DATE_EPOCH"))
`

Did you look at the git repository on salsa.debian.org?  It only
contains the debian/ directory, so SOURCE_DATE_EPOCH does indeed not
show up there…

Cheers,
   Sven

Bug#864082: fontconfig: please make the cache files reproducible

2018-09-26 Thread Chris Lamb 
Hi Sven,

> It seems that 2.13.1-1 was that release, can you confirm that?

Thanks for chiming in.

Given that this version does not contain the string SOURCE_DATE_EPOCH,
this was probably another release and/or this should remain open.


Regards,

-- 
  ,''`.
 : :'  : Chris Lamb
 `. `'`  la...@debian.org / chris-lamb.co.uk
   `-

Bug#864082: fontconfig: please make the cache files reproducible

2018-09-26 Thread Sven Joachim 
On 2018-05-18 08:42 +0100, Chris Lamb wrote:

> Hi,
>
>> fontconfig: please make the cache files reproducible
>
> This was merged into the upstream Git repository - would it be
> possible to make another Debian release with this change? :)

It seems that 2.13.1-1 was that release, can you confirm that?

Just flying by - apparently the new fontconfig upstream release has
fixed several bugs in the BTS, but the Debian changelog did not mention
any.

Cheers,
   Sven

Bug#864082: fontconfig: please make the cache files reproducible

2018-08-24 Thread Chris Lamb 
Dear Keith,

> > Gentle ping on this? :) Would love to see this Tails-related work
> > in Debian!
> 
> I was stalling for an upstream release with this patch; it looks like
> that shouldn't be more than a month or two from now. Any particular
> reason for urgency here?

Somewhat; it's currently causing buster-based builds for Tails to be
unreproducible.

This means that the in-depth results are being ignored (hey "it's
already unreproducible!") and thus other issues may pile up, causing
scheduling/fixing issues later on.


Best wishes,

-- 
  ,''`.
 : :'  : Chris Lamb
 `. `'`  la...@debian.org / chris-lamb.co.uk
   `-

Bug#864082: fontconfig: please make the cache files reproducible

2018-08-05 Thread Chris Lamb 
Dear Keith,

> > Gentle ping on this? :) Would love to see this Tails-related work
> > in Debian!
> 
> I was stalling for an upstream release with this patch; it looks like
> that shouldn't be more than a month or two from now.

Heh, it seems like fontconfig is perpetually in this state alas. ;)

> Any particular reason for urgency here?

Not especially but it's always nice to reduce the package differences
between in Tails and Debian; the philosophy is very much in that
direction rather than "just" throwing patches over the wall and hoping
they land.


Regards,

-- 
  ,''`.
 : :'  : Chris Lamb
 `. `'`  la...@debian.org / chris-lamb.co.uk
   `-

Bug#864082: fontconfig: please make the cache files reproducible

2018-08-05 Thread Keith Packard 
Chris Lamb  writes:

> Chris Lamb wrote:
>
>> This was merged into the upstream Git repository - would it be
>> possible to make another Debian release with this change? :)
>
> Gentle ping on this? :) Would love to see this Tails-related work
> in Debian!

I was stalling for an upstream release with this patch; it looks like
that shouldn't be more than a month or two from now. Any particular
reason for urgency here?

-- 
-keith


signature.asc
Description: PGP signature

Bug#864082: fontconfig: please make the cache files reproducible

2018-08-04 Thread Chris Lamb 
Chris Lamb wrote:

> This was merged into the upstream Git repository - would it be
> possible to make another Debian release with this change? :)

Gentle ping on this? :) Would love to see this Tails-related work
in Debian!


Best wishes,

-- 
  ,''`.
 : :'  : Chris Lamb
 `. `'`  la...@debian.org / chris-lamb.co.uk
   `-

Bug#864082: fontconfig: please make the cache files reproducible

2018-05-24 Thread Chris Lamb 
Chris Lamb wrote:

> This was merged into the upstream Git repository - would it be
> possible to make another Debian release with this change? :)

Gentle ping on this? :)


Best wishes,

-- 
  ,''`.
 : :'  : Chris Lamb
 `. `'`  la...@debian.org / chris-lamb.co.uk
   `-

Bug#864082: fontconfig: please make the cache files reproducible

2018-05-18 Thread Chris Lamb 
Hi,

> fontconfig: please make the cache files reproducible

This was merged into the upstream Git repository - would it be
possible to make another Debian release with this change? :)


Best wishes,

-- 
  ,''`.
 : :'  : Chris Lamb
 `. `'`  la...@debian.org / chris-lamb.co.uk
   `-

Bug#864082: fontconfig: please make the cache files reproducible

2018-05-05 Thread Chris Lamb 
forwarded 864082 
https://lists.freedesktop.org/archives/fontconfig/2018-May/006271.html
thanks

This is now *really* on the upstream mailing list... :)

  https://lists.freedesktop.org/archives/fontconfig/2018-May/006271.html


Regards,

-- 
  ,''`.
 : :'  : Chris Lamb
 `. `'`  la...@debian.org / chris-lamb.co.uk
   `-

Bug#864082: fontconfig: please make the cache files reproducible

2018-05-03 Thread Chris Lamb 
Hi Keith,

> I think it would be useful for me to understand when and where the cache
> files end up being part of a build product

The idea is that two entirely separate builds of an essentially-
vanilla Debian-based ISO end up being bit-for-bit identical.

Currently that is not occurring as, for example:

  /var/cache/fontconfig/04aabc0a78ac019cf9454389977116d2-le64.cache-7

.. in a "first" build has different contents to the one in a
second varying based on the mtime of the source directories (which
were naturally created at different times).

> I can imagine a special flag to fc-cache that would mark the cache
> files for this use. I feel that this would solve the problem in a
> better way.

As outlined above, the problem is more that the cache files
themselves are varying on each creation. It is also unclear how
this special flag would be activated via, essentially, a call to
"apt install fontpkg". :)


Regards,

-- 
  ,''`.
 : :'  : Chris Lamb
 `. `'`  la...@debian.org / chris-lamb.co.uk
   `-

Bug#864082: fontconfig: please make the cache files reproducible

2018-05-03 Thread Keith Packard 
Chris Lamb  writes:

> Hi Keith,
>
>> > +source_date_epoch = getenv("SOURCE_DATE_EPOCH");
>> 
>> Could this work as a build-time value in the library instead of a
>> run-time environment variable?
>
> Unfortunately not. Imagine the situation where we are installing
> font packages in a chroot that will eventually end up as, for
> example, an .ISO: in this case, we are running fc-cache at runtime
> (in Debian's case, via the dpkg trigger).

Thanks for the explanation.

I think it would be useful for me to understand when and where the cache
files end up being part of a build product and then figuring out what
the right solution is in each case, rather than an environment variable
kludge of this nature.

For instance, in the case described above, the ISO is read-only in use,
and so the cache file contents *cannot* be out of date, and should
always be used with no need to even check the timestamps on
directories. I can imagine a special flag to fc-cache that would
mark the cache files for this use. I feel that this would solve the
problem in a better way.

-- 
-keith


signature.asc
Description: PGP signature

Bug#864082: fontconfig: please make the cache files reproducible

2018-05-03 Thread Chris Lamb 
Hi Keith,

> > +source_date_epoch = getenv("SOURCE_DATE_EPOCH");
> 
> Could this work as a build-time value in the library instead of a
> run-time environment variable?

Unfortunately not. Imagine the situation where we are installing
font packages in a chroot that will eventually end up as, for
example, an .ISO: in this case, we are running fc-cache at runtime
(in Debian's case, via the dpkg trigger).


Regards,

-- 
  ,''`.
 : :'  : Chris Lamb
 `. `'`  la...@debian.org / chris-lamb.co.uk
   `-

Bug#864082: fontconfig: please make the cache files reproducible

2018-05-03 Thread Keith Packard 
Chris Lamb  writes:


> +source_date_epoch = getenv("SOURCE_DATE_EPOCH");

Could this work as a build-time value in the library instead of a
run-time environment variable?

-- 
-keith


signature.asc
Description: PGP signature

Bug#864082: fontconfig: please make the cache files reproducible

2018-05-02 Thread Chris Lamb 
Chris Lamb wrote:

> A better patch is in-progress on the upstream mailing list, also
> attached here.

It should eventually appear here, I assume I'm stuck in a
moderation queue or similar:

  https://lists.freedesktop.org/archives/fontconfig/2018-May/thread.html


Regards,

-- 
  ,''`.
 : :'  : Chris Lamb
 `. `'`  la...@debian.org / chris-lamb.co.uk
   `-

Bug#864082: fontconfig: please make the cache files reproducible

2018-05-02 Thread Chris Lamb 
Chris Lamb wrote:

> Patch attached.

A better patch is in-progress on the upstream mailing list, also
attached here.


Regards,

-- 
  ,''`.
 : :'  : Chris Lamb
 `. `'`  la...@debian.org / chris-lamb.co.uk
   `-
From ccc5da9ed364df1334a3ff172d89f852d199955b Mon Sep 17 00:00:00 2001
From: Chris Lamb 
Date: Sat, 28 Apr 2018 12:21:24 -0700
Subject: [PATCH] Ensure cache checksums are determinstic

Whilst working on the Reproducible Builds[0] effort, we noticed that
fontconfig generates unreproducible cache files.

This is due to fc-cache uses the modification timestamps of each
directory in the "checksum" and "checksum_nano" members of the _FcCache
struct. This is so that it can identify which cache files are valid
and/or require regeneration.

This patch changes the behaviour of the checksum calculations to prefer
the value of the SOURCE_DATE_EPOCH[1] environment variable over the
directory's own mtime. This variable can then be exported by build
systems to ensure reproducible output.

If SOURCE_DATE_EPOCH is not set or is newer than the mtime of the
directory, the existing behaviour is unchanged.

This work was sponsored by Tails[2].

 [0] https://reproducible-builds.org/
 [1] https://reproducible-builds.org/specs/source-date-epoch/
 [2] https://tails.boum.org/
---
 src/fccache.c | 55 +--
 1 file changed, 49 insertions(+), 6 deletions(-)

diff --git a/src/fccache.c b/src/fccache.c
index 7abb750..e74d902 100644
--- a/src/fccache.c
+++ b/src/fccache.c
@@ -989,6 +989,51 @@ FcDirCacheLoadFile (const FcChar8 *cache_file, struct stat *file_stat)
 return cache;
 }
 
+static int
+FcDirChecksum (struct stat *statb) {
+int			ret = (int) statb->st_mtime;
+char		*endptr;
+char		*source_date_epoch;
+unsigned long long	epoch;
+
+source_date_epoch = getenv("SOURCE_DATE_EPOCH");
+if (source_date_epoch) {
+	epoch = strtoull(source_date_epoch, , 10);
+
+	if (endptr == source_date_epoch)
+	fprintf (stderr,
+		 "Fontconfig: SOURCE_DATE_EPOCH invalid\n");
+	else if ((errno == ERANGE && (epoch == ULLONG_MAX || epoch == 0))
+		|| (errno != 0 && epoch == 0))
+	fprintf (stderr,
+		 "Fontconfig: SOURCE_DATE_EPOCH: strtoull: %s: %llu\n",
+		 strerror(errno), epoch);
+	else if (*endptr != '\0')
+	fprintf (stderr,
+		 "Fontconfig: SOURCE_DATE_EPOCHh has trailing garbage\n");
+	else if (epoch > ULONG_MAX)
+	fprintf (stderr,
+		 "Fontconfig: SOURCE_DATE_EPOCH must be <= %lu but saw: %llu\n",
+		 ULONG_MAX, epoch);
+	else if (epoch < ret)
+	/* Only override if directory is newer */
+	ret = (int) epoch;
+}
+
+return ret;
+}
+
+static int64_t
+FcDirChecksumNano (struct stat *statb) {
+/* No nanosecond component to parse */
+if (getenv("SOURCE_DATE_EPOCH"))
+	return 0;
+#ifdef HAVE_STRUCT_STAT_ST_MTIM
+return statb->st_mtim.tv_nsec;
+#endif
+return 0;
+}
+
 /*
  * Validate a cache file by reading the header and checking
  * the magic number and the size field
@@ -1007,10 +1052,10 @@ FcDirCacheValidateHelper (FcConfig *config, int fd, struct stat *fd_stat, struct
 	ret = FcFalse;
 else if (fd_stat->st_size != c.size)
 	ret = FcFalse;
-else if (c.checksum != (int) dir_stat->st_mtime)
+else if (c.checksum != FcDirChecksum(dir_stat))
 	ret = FcFalse;
 #ifdef HAVE_STRUCT_STAT_ST_MTIM
-else if (c.checksum_nano != dir_stat->st_mtim.tv_nsec)
+else if (c.checksum_nano != FcDirChecksumNano(dir_stat))
 	ret = FcFalse;
 #endif
 return ret;
@@ -1086,10 +1131,8 @@ FcDirCacheBuild (FcFontSet *set, const FcChar8 *dir, struct stat *dir_stat, FcSt
 cache->magic = FC_CACHE_MAGIC_ALLOC;
 cache->version = FC_CACHE_VERSION_NUMBER;
 cache->size = serialize->size;
-cache->checksum = (int) dir_stat->st_mtime;
-#ifdef HAVE_STRUCT_STAT_ST_MTIM
-cache->checksum_nano = dir_stat->st_mtim.tv_nsec;
-#endif
+cache->checksum = FcDirChecksum(dir_stat);
+cache->checksum_nano = FcDirChecksumNano(dir_stat);
 
 /*
  * Serialize directory name
-- 
2.17.0

Bug#864082: fontconfig: please make the cache files reproducible

2017-06-04 Thread Chris Lamb 
Source: fontconfig
Version: 2.12.1-0.1
Severity: wishlist
Tags: patch
User: reproducible-bui...@lists.alioth.debian.org
Usertags: timestamps
X-Debbugs-Cc: reproducible-b...@lists.alioth.debian.org

Hi,

Whilst working on the Reproducible Builds effort [0], we noticed that
fontconfig generates non-reproducible cache files under
/var/cache/fontconfig.

This is because fontconfig embeds the mtime of each font directory
in a "checksum" member of a "_FcCache" struct. This is so that it
can identify which cache files remain valid and/or require
regeneration.

We therefore "clamp" the mtimes of font directories to SOURCE_DATE_EPOCH
prior to calling fc-cache to avoid these non-deterministic values
appearing in the files themselves. This is safe as we now force
regeneration in subsequent fc-cache calls with -f.

(We can't just replace the checksum value with SOURCE_DATE_EPOCH as it
will result in fontconfig believing the cache to be outdated, defeating
the entire point of generating them in the first place.)

This work was sponsored by Tails[1].

Patch attached.


 [0] https://reproducible-builds.org/
 [1] https://tails.boum.org/


Regards,

-- 
  ,''`.
 : :'  : Chris Lamb
 `. `'`  la...@debian.org / chris-lamb.co.uk
   `-
diff --git a/debian/fontconfig.postinst b/debian/fontconfig.postinst
index ad7ac19..dfba70e 100644
--- a/debian/fontconfig.postinst
+++ b/debian/fontconfig.postinst
@@ -2,10 +2,28 @@
 
 set -e
 
+if [ -n "$SOURCE_DATE_EPOCH" ]; then
+  # fontconfig embeds the mtime of each font directory in a "checksum" member
+  # of a "_FcCache" struct. This is so that it can identify which cache files
+  # remain valid and/or require regeneration.
+  #
+  # We therefore "clamp" the mtimes of font directories to SOURCE_DATE_EPOCH
+  # prior to calling fc-cache to avoid these non-deterministic values appearing
+  # in the files themselves. This is safe as we force regeneration in
+  # subsequent fc-cache calls with -f.
+  #
+  # (We can't just replace the checksum value with SOURCE_DATE_EPOCH as it will
+  # result in fontconfig believing the cache to be outdated, defeating the
+  # entire point of generating them in the first place.
+  fc-cache -s --list-dirs | \
+xargs -I{} find {} -type d -follow -newermt "@$SOURCE_DATE_EPOCH" -print0 
2>/dev/null | \
+xargs -0r touch --date="@$SOURCE_DATE_EPOCH"
+fi
+
 if [ "$1" = triggered ]; then
   # Force regeneration of all fontconfig cache files.
   mkdir -p /var/cache/fontconfig
-  fc-cache -s -v 1>/var/log/fontconfig.log 2>&1 || printf "fc-cache 
failed.\nSee /var/log/fontconfig.log for more information.\n"
+  fc-cache -s -f -v 1>/var/log/fontconfig.log 2>&1 || printf "fc-cache 
failed.\nSee /var/log/fontconfig.log for more information.\n"
   exit 0
 fi
 
diff --git a/fc-cache/fc-cache.1 b/fc-cache/fc-cache.1
index e514779..f5a733d 100644
--- a/fc-cache/fc-cache.1
+++ b/fc-cache/fc-cache.1
@@ -4,7 +4,7 @@
 fc-cache \- build font information cache files
 .SH SYNOPSIS
 .sp
-\fBfc-cache\fR [ \fB-EfrsvVh\fR ]  [ \fB--error-on-no-fonts\fR ]  [ 
\fB--force\fR ]  [ \fB--really-force\fR ]  [ \fB [ -y \fIdir\fB ]  [ --sysroot 
\fIdir\fB ] \fR ]  [ \fB--system-only\fR ]  [ \fB--verbose\fR ]  [ 
\fB--version\fR ]  [ \fB--help\fR ]  [ \fB\fIdir\fB\fR\fI...\fR ] 
+\fBfc-cache\fR [ \fB-EfrsvVh\fR ]  [ \fB--error-on-no-fonts\fR ]  [ 
\fB--force\fR ]  [ \fB--really-force\fR ]  [ \fB [ -y \fIdir\fB ]  [ --sysroot 
\fIdir\fB ] \fR ]  [ \fB--system-only\fR ]  [ \fB--list-dirs\fR ]  [ 
\fB--verbose\fR ]  [ \fB--version\fR ]  [ \fB--help\fR ]  [ 
\fB\fIdir\fB\fR\fI...\fR ] 
 .SH "DESCRIPTION"
 .PP
 \fBfc-cache\fR scans the font directories on
@@ -44,6 +44,9 @@ Erase all existing cache files and rescan.
 Only scan system-wide directories, omitting the places
 located in the user's home directory.
 .TP
+\fB-l\fR
+Only list directories, don't regenerate anything.
+.TP
 \fB-v\fR
 Display status information while busy.
 .TP
diff --git a/fc-cache/fc-cache.c b/fc-cache/fc-cache.c
index 0336073..fc5ff07 100644
--- a/fc-cache/fc-cache.c
+++ b/fc-cache/fc-cache.c
@@ -70,6 +70,7 @@ const struct option longopts[] = {
 {"really-force", 0, 0, 'r'},
 {"sysroot", required_argument, 0, 'y'},
 {"system-only", 0, 0, 's'},
+{"list-dirs", 0, 0, 'l'},
 {"version", 0, 0, 'V'},
 {"verbose", 0, 0, 'v'},
 {"help", 0, 0, 'h'},
@@ -87,10 +88,10 @@ usage (char *program, int error)
 {
 FILE *file = error ? stderr : stdout;
 #if HAVE_GETOPT_LONG
-fprintf (file, "usage: %s [-EfrsvVh] [-y SYSROOT] [--error-on-no-fonts] 
[--force|--really-force] [--sysroot=SYSROOT] [--system-only] [--verbose] 
[--version] [--help] [dirs]\n",
+fprintf (file, "usage: %s [-EfrslvVh] [-y SYSROOT] [--error-on-no-fonts] 
[--force|--really-force] [--sysroot=SYSROOT] [--system-only] [--list-dirs] 
[--verbose] [--version] [--help] [dirs]\n",
 program);
 #else
-fprintf (file, "usage: %s [-EfrsvVh] [-y SYSROOT] [dirs]\n",
+fprintf (file, "usage: %s