Bug#828683: mc: please make the build reproducible
On 2022-12-22, Vagrant Cascadian wrote: > On 2022-10-06, Vagrant Cascadian wrote: >> The attached alternate implements this for mc by touching the potential >> files before running configure with a consistent timestamp. >> >> According to my local tests, applying this patch should make mc build >> reproducibly once it lands in testing/bookworm! There are outstanding >> build path issues tested in unstable and experimental. > ... >> From 4e69587954d29ec6bfc7d85b4b618724b16b840e Mon Sep 17 00:00:00 2001 >> From: Vagrant Cascadian >> Date: Thu, 6 Oct 2022 19:25:11 + >> Subject: [PATCH 5/5] debian/rules: Ensure consistent timestamp on manpages. >> >> The upstream build system uses the file modification time of the .in >> file for the manpage to embed into the generatated manpage, but if >> debian/patches modify the .in files, the timestamp is updated, >> resulting in builds performed in different months or years embedding a >> different date. >> --- >> debian/rules | 2 ++ >> 1 file changed, 2 insertions(+) >> >> diff --git a/debian/rules b/debian/rules >> index c633a73..bf8eb5e 100755 >> --- a/debian/rules >> +++ b/debian/rules >> @@ -25,6 +25,8 @@ override_dh_autoreconf: >> # AWK="awk" is inheritance of 4.7.* series, see >> http://bugs.debian.org/499723 >> # might be still necessary for extfs scripts >> override_dh_auto_configure: >> +# Ensure reproducible timestamp on manpages >> +touch -d@$(SOURCE_DATE_EPOCH) doc/man/*.1.in doc/man/*/*.1.in >> dh_auto_configure -- AWK="awk" X11_WWW="x-www-browser" \ >> --libexecdir='/usr/lib' \ >> --with-x \ >> -- >> 2.37.2 > > I Intend to NMU with this patch on the 29th (with a 10-day delay) unless > I hear otherwise. A little behind schedule, and a little more conservative patch (only touching the currently patched mcedit manpage)... Uploaded NMU to DELAYED/10: diff -Nru mc-4.8.28/debian/changelog mc-4.8.28/debian/changelog --- mc-4.8.28/debian/changelog 2022-04-02 15:10:15.0 -0700 +++ mc-4.8.28/debian/changelog 2023-01-10 09:50:36.0 -0800 @@ -1,3 +1,11 @@ +mc (3:4.8.28-1.1) unstable; urgency=medium + + * Non-maintainer upload. + * debian/rules: Ensure consistent timestamp on mcedit manpage. +(Closes: #828683) + + -- Vagrant Cascadian Tue, 10 Jan 2023 09:50:36 -0800 + mc (3:4.8.28-1) unstable; urgency=medium * New upstream release. diff -Nru mc-4.8.28/debian/rules mc-4.8.28/debian/rules --- mc-4.8.28/debian/rules 2018-06-05 21:11:33.0 -0700 +++ mc-4.8.28/debian/rules 2023-01-10 09:49:58.0 -0800 @@ -25,6 +25,9 @@ # AWK="awk" is inheritance of 4.7.* series, see http://bugs.debian.org/499723 # might be still necessary for extfs scripts override_dh_auto_configure: + # Ensure reproducible timestamp on mcedit manpage, which is + # patched from debian/patches + touch -d@$(SOURCE_DATE_EPOCH) doc/man/mcedit.1.in dh_auto_configure -- AWK="awk" X11_WWW="x-www-browser" \ --libexecdir='/usr/lib' \ --with-x \ signature.asc Description: PGP signature
Bug#828683: mc: please make the build reproducible
On Thu, 22 Dec 2022, Vagrant Cascadian wrote: I Intend to NMU with this patch on the 29th (with a 10-day delay) unless I hear otherwise. I have no objections to that; please proceed, if you believe that this is the right thing to do. Your graceful heads up is appreciated. I was just trying to explain, that in my opinion the source of the problem is resetting the timestamps while applying distro patches, so maybe it's worth looking at fixing the source instead of downstream consequences, and I'm unwilling to do something upstream to accomodate a distro issue, but at distro level, you decide what is the best way to go on. -- Sincerely yours, Yury V. Zaytsev
Bug#828683: mc: please make the build reproducible
On 2022-10-06, Vagrant Cascadian wrote: > The attached alternate implements this for mc by touching the potential > files before running configure with a consistent timestamp. > > According to my local tests, applying this patch should make mc build > reproducibly once it lands in testing/bookworm! There are outstanding > build path issues tested in unstable and experimental. ... > From 4e69587954d29ec6bfc7d85b4b618724b16b840e Mon Sep 17 00:00:00 2001 > From: Vagrant Cascadian > Date: Thu, 6 Oct 2022 19:25:11 + > Subject: [PATCH 5/5] debian/rules: Ensure consistent timestamp on manpages. > > The upstream build system uses the file modification time of the .in > file for the manpage to embed into the generatated manpage, but if > debian/patches modify the .in files, the timestamp is updated, > resulting in builds performed in different months or years embedding a > different date. > --- > debian/rules | 2 ++ > 1 file changed, 2 insertions(+) > > diff --git a/debian/rules b/debian/rules > index c633a73..bf8eb5e 100755 > --- a/debian/rules > +++ b/debian/rules > @@ -25,6 +25,8 @@ override_dh_autoreconf: > # AWK="awk" is inheritance of 4.7.* series, see http://bugs.debian.org/499723 > # might be still necessary for extfs scripts > override_dh_auto_configure: > + # Ensure reproducible timestamp on manpages > + touch -d@$(SOURCE_DATE_EPOCH) doc/man/*.1.in doc/man/*/*.1.in > dh_auto_configure -- AWK="awk" X11_WWW="x-www-browser" \ > --libexecdir='/usr/lib' \ > --with-x \ > -- > 2.37.2 I Intend to NMU with this patch on the 29th (with a 10-day delay) unless I hear otherwise. live well, vagrant signature.asc Description: PGP signature
Bug#828683: mc: please make the build reproducible
On 2016-06-27, Yury V. Zaytsev wrote:
> On Mon, 27 Jun 2016, Reiner Herrmann wrote:
Going through some old outstanding reproducible builds related bugs...
>> You are right, the mtime of the file is used for the manpage timestamp.
>> But there is still a patch modifying the manpage:
>> mcedit_full_path.patch. When this patch is upstreamed, the
>> SOURCE_DATE_EPOCH patch is not needed, though it would still help when
>> the manpages are patched.
>
> Thank you for the confirmation that my guesses are correct!
>
> However, I'm hesitant to apply SOURCE_DATE_EPOCH to anything that looks
> like a nail. I believe that patching of upstream sources by Debian is such
> an oft occurring situation, that I would rather look for at solution at
> the packaging level. After all, builds from *upstream* sources are
> *already* reproducible, in this case, it's the Debian build procedure that
> is injecting randomness here.
I am biased, but I do think using SOURCE_DATE_EPOCH could still be a
reasonable approach upstream. There are some examples of how to use this
in various languages:
https://reproducible-builds.org/docs/source-date-epoch/
I did notice some things in the upstream code
doc/man/date-of-man-include.am:
MAN_DATE_CMD = \
LC_ALL=$(DATE_LANG) @PERL@ -CS -MPOSIX -e '\
@fi=lstat("'$${MAN_FILE}'"); \
print POSIX::strftime("$(DATE_FORMAT)", localtime($$fi[9]));'
2>/dev/null
This uses localtime, which could be affected by timezone differences. To
my surprise, it doesn't appear to affect the results, as only the
patched manpage is affected:
https://tests.reproducible-builds.org/debian/rb-pkg/bookworm/amd64/diffoscope-results/mc.html
Setting LC_ALL explicitly avoids variations due to the build
environment's locale(yay!), although the specified locales are not
guaranteed to be available (adding locales-all to build-depends could
fix that).
Again, surprisingly, this has not appeared to affect the results for the
locales tested on tests.reproducible-builds.org, though that may mean
the default locale is used even for the translated pages...
> For instance, it seems to me that it is only logical to set the mtime of
> patched files to the mtime of the last patch that touched them, and this
> will make the source mtime dependent builds fully reproducible.
The attached alternate implements this for mc by touching the potential
files before running configure with a consistent timestamp.
According to my local tests, applying this patch should make mc build
reproducibly once it lands in testing/bookworm! There are outstanding
build path issues tested in unstable and experimental.
It doesn't address the larger issue of files modified by debian/patches
having the current timestamp, which is a much more complicated
intersection of issues.
live well,
vagrant
From 4e69587954d29ec6bfc7d85b4b618724b16b840e Mon Sep 17 00:00:00 2001
From: Vagrant Cascadian
Date: Thu, 6 Oct 2022 19:25:11 +
Subject: [PATCH 5/5] debian/rules: Ensure consistent timestamp on manpages.
The upstream build system uses the file modification time of the .in
file for the manpage to embed into the generatated manpage, but if
debian/patches modify the .in files, the timestamp is updated,
resulting in builds performed in different months or years embedding a
different date.
---
debian/rules | 2 ++
1 file changed, 2 insertions(+)
diff --git a/debian/rules b/debian/rules
index c633a73..bf8eb5e 100755
--- a/debian/rules
+++ b/debian/rules
@@ -25,6 +25,8 @@ override_dh_autoreconf:
# AWK="awk" is inheritance of 4.7.* series, see http://bugs.debian.org/499723
# might be still necessary for extfs scripts
override_dh_auto_configure:
+ # Ensure reproducible timestamp on manpages
+ touch -d@$(SOURCE_DATE_EPOCH) doc/man/*.1.in doc/man/*/*.1.in
dh_auto_configure -- AWK="awk" X11_WWW="x-www-browser" \
--libexecdir='/usr/lib' \
--with-x \
--
2.37.2
signature.asc
Description: PGP signature
Bug#828683: [Reproducible-builds] [Pkg-mc-devel] Bug#828683: mc: please make the build reproducible
On Mon, Jun 27, 2016 at 10:39:52AM +0200, Yury V. Zaytsev wrote: > For instance, it seems to me that it is only logical to set the mtime of > patched files to the mtime of the last patch that touched them, and this > will make the source mtime dependent builds fully reproducible. > > Have you already thought of that, and if yes, what are the counter arguments > against this approach? Yes, we already thought of that, unfortunately there will be issues, if the timestamp is reset after patching. From dpkg-source(1) for 3.0 (quilt) format: > The timestamp of all patched files is reset to the extraction time of > the source package (this avoids timestamp skews leading to problems > when autogenerated files are patched). signature.asc Description: Digital signature
Bug#828683: [Pkg-mc-devel] Bug#828683: mc: please make the build reproducible
Hi Reiner, On Mon, 27 Jun 2016, Reiner Herrmann wrote: You are right, the mtime of the file is used for the manpage timestamp. But there is still a patch modifying the manpage: mcedit_full_path.patch. When this patch is upstreamed, the SOURCE_DATE_EPOCH patch is not needed, though it would still help when the manpages are patched. Thank you for the confirmation that my guesses are correct! However, I'm hesitant to apply SOURCE_DATE_EPOCH to anything that looks like a nail. I believe that patching of upstream sources by Debian is such an oft occurring situation, that I would rather look for at solution at the packaging level. After all, builds from *upstream* sources are *already* reproducible, in this case, it's the Debian build procedure that is injecting randomness here. For instance, it seems to me that it is only logical to set the mtime of patched files to the mtime of the last patch that touched them, and this will make the source mtime dependent builds fully reproducible. Have you already thought of that, and if yes, what are the counter arguments against this approach? -- Sincerely yours, Yury V. Zaytsev
Bug#828683: [Pkg-mc-devel] Bug#828683: mc: please make the build reproducible
On Sun, Jun 26, 2016 at 10:21:49PM +0200, Yury V. Zaytsev wrote: > On Sun, 26 Jun 2016, Reiner Herrmann wrote: > > >While working on the "reproducible builds" effort [1], we have noticed > >that mc could not be built reproducibly. It embeds the current date into > >the mcedit manpage during build. > > Ok, I think that I can now see what went wrong here: Debian used to patch > mcedit manpage, and so, mtime of the manpage file was set to the current > time during the build. Our build system faithfully hardcoded this into the > resulting man file. > > Since then, however, I've upstreamed the patch and it was removed from > 3:4.8.17-1, so I'm really wondering why it is still not working for you... You are right, the mtime of the file is used for the manpage timestamp. But there is still a patch modifying the manpage: mcedit_full_path.patch. When this patch is upstreamed, the SOURCE_DATE_EPOCH patch is not needed, though it would still help when the manpages are patched. signature.asc Description: Digital signature
Bug#828683: [Pkg-mc-devel] Bug#828683: mc: please make the build reproducible
On Sun, 26 Jun 2016, Reiner Herrmann wrote: While working on the "reproducible builds" effort [1], we have noticed that mc could not be built reproducibly. It embeds the current date into the mcedit manpage during build. Ok, I think that I can now see what went wrong here: Debian used to patch mcedit manpage, and so, mtime of the manpage file was set to the current time during the build. Our build system faithfully hardcoded this into the resulting man file. Since then, however, I've upstreamed the patch and it was removed from 3:4.8.17-1, so I'm really wondering why it is still not working for you... -- Sincerely yours, Yury V. Zaytsev
Bug#828683: [Pkg-mc-devel] Bug#828683: mc: please make the build reproducible
Hi Reiner, On Sun, 26 Jun 2016, Reiner Herrmann wrote: Yes, it is standardized [1] and already supported by a lot of build tools [2], e.g. even by gcc. Other distributions and FreeBSD are currently also in the process of adopting it. Thank you for the clarifications! Maybe you could consider including these links in the original reports to mitigate such follow-up questions... I've had a look at our build system, and my impression is that your patch should not be necessary. Please have a look at doc/man/date-of-man-include.am and in particular at MAN_DATE_CMD variable and following targets. It encodes mtime of the source man file in the output, rather than the build date, so, as long as the source date is not modified, the build should be reproducible. I believe that the lines that you have patched are currently unused and should be simply removed. Am I missing something? -- Sincerely yours, Yury V. Zaytsev
Bug#828683: [Pkg-mc-devel] Bug#828683: mc: please make the build reproducible
Hi Reiner, On Sun, 26 Jun 2016, Reiner Herrmann wrote: While working on the "reproducible builds" effort [1], we have noticed that mc could not be built reproducibly. It embeds the current date into the mcedit manpage during build. That's my fault: the dates were originally entered by hand, which, of course, soon enough resulted in that they were no longer current, so I campaigned for them to be set automatically during the build. The attached patch uses SOURCE_DATE_EPOCH as a deterministic timestamp instead. I wouldn't mind upstreaming this patch as long as SOURCE_DATE_EPOCH is at least halfway standardized. Could you please tell me whether this is Debian-specific, or other distros are also adopting this convention? Many thanks! -- Sincerely yours, Yury V. Zaytsev
Bug#828683: [Pkg-mc-devel] Bug#828683: mc: please make the build reproducible
On Sun, Jun 26, 2016 at 09:00:50PM +0200, Yury V. Zaytsev wrote: > >The attached patch uses SOURCE_DATE_EPOCH as a deterministic timestamp > >instead. > > I wouldn't mind upstreaming this patch as long as SOURCE_DATE_EPOCH is at > least halfway standardized. Could you please tell me whether this is > Debian-specific, or other distros are also adopting this convention? Thanks for intending to upstream it! Yes, it is standardized [1] and already supported by a lot of build tools [2], e.g. even by gcc. Other distributions and FreeBSD are currently also in the process of adopting it. [1]: https://reproducible-builds.org/specs/source-date-epoch/ [2]: https://wiki.debian.org/ReproducibleBuilds/TimestampsProposal#Reading_the_variable signature.asc Description: Digital signature
Bug#828683: mc: please make the build reproducible
Source: mc
Version: 3:4.8.17-1
Severity: wishlist
Tags: patch
User: reproducible-bui...@lists.alioth.debian.org
Usertags: timestamps
X-Debbugs-Cc: reproducible-bui...@lists.alioth.debian.org
Hi!
While working on the "reproducible builds" effort [1], we have noticed
that mc could not be built reproducibly.
It embeds the current date into the mcedit manpage during build.
The attached patch uses SOURCE_DATE_EPOCH as a deterministic
timestamp instead.
Regards,
Reiner
[1]: https://wiki.debian.org/ReproducibleBuilds
diff --git a/debian/patches/reproducible_man_date.patch b/debian/patches/reproducible_man_date.patch
new file mode 100644
index 000..5bcac25
--- /dev/null
+++ b/debian/patches/reproducible_man_date.patch
@@ -0,0 +1,13 @@
+--- a/configure.ac
b/configure.ac
+@@ -428,7 +428,9 @@
+ dnl Documentation
+ dnl
+
+-MAN_DATE="$(LC_ALL=C date "+%B %Y")"
++DATE_FMT="%B %Y"
++SOURCE_DATE_EPOCH="${SOURCE_DATE_EPOCH:-$(date +%s)}"
++MAN_DATE=$(LC_ALL=C date -u -d "@$SOURCE_DATE_EPOCH" "+$DATE_FMT" 2>/dev/null || LC_ALL=C date -u -r "$SOURCE_DATE_EPOCH" "+$DATE_FMT" 2>/dev/null || LC_ALL=C date -u "+$DATE_FMT")
+ AC_SUBST(MAN_DATE)
+
+ dnl Determine which help translations we want to install.
diff --git a/debian/patches/series b/debian/patches/series
index 861e26f..81c7c50 100644
--- a/debian/patches/series
+++ b/debian/patches/series
@@ -14,3 +14,4 @@ mcedit_full_path.patch
mcedit_group_undo.patch
ext_run-mailcap.patch
ext_use_default_editor.patch
+reproducible_man_date.patch
