Re: [PATCH 1/3] Documentation: allow overriding timestamps of generated asciidoc

[Jonathan Nieder]

Anders Kaseorg wrote:
> On Tue, 21 Nov 2017, Jonathan Nieder wrote:

>> http://asciidoc.org/CHANGELOG.html is stale but asciidoc still seems
>> to be getting changes at https://github.com/asciidoc/asciidoc.  I
>> wonder how difficult it would be to add any required SOURCE_DATE_EPOCH
>> support there.
>
> In fact I already did (https://github.com/asciidoc/asciidoc/pull/106),
> which is why I’d been holding off on trying to upstream this Git patch.
> The trouble was, the AsciiDoc developers had not been cutting new releases
> “because nobody knows how”
> (https://github.com/asciidoc/asciidoc/issues/103#issuecomment-322077321).
> However, it looks like AsciiDoc 8.6.10 was recently tagged and Debian got
> a 8.6.10-1 package yesterday, so I guess that trouble has been quietly
> resolved.

Ah, lovely.  I'll add a build-time dependency on that version to the
Debian package.

Junio, please disregard this patch (patch 1/3).

Thanks,
Jonathan

Re: [PATCH 1/3] Documentation: allow overriding timestamps of generated asciidoc

[Junio C Hamano]

Anders Kaseorg  writes:

> That should make this Git patch unnecessary.  (You’re of course still 
> welcome to take it if you think build reproducibility with old AsciiDoc 
> versions is worthwhile.)

Thanks.  

I've queued these three only so that I won't lose track, but will
not hastily merge them down (yet) until I hear from people.

Re: [PATCH 1/3] Documentation: allow overriding timestamps of generated asciidoc

[Anders Kaseorg]

On Tue, 21 Nov 2017, Jonathan Nieder wrote:
> http://asciidoc.org/CHANGELOG.html is stale but asciidoc still seems
> to be getting changes at https://github.com/asciidoc/asciidoc.  I
> wonder how difficult it would be to add any required SOURCE_DATE_EPOCH
> support there.

In fact I already did (https://github.com/asciidoc/asciidoc/pull/106), 
which is why I’d been holding off on trying to upstream this Git patch.  
The trouble was, the AsciiDoc developers had not been cutting new releases 
“because nobody knows how” 
(https://github.com/asciidoc/asciidoc/issues/103#issuecomment-322077321). 
However, it looks like AsciiDoc 8.6.10 was recently tagged and Debian got 
a 8.6.10-1 package yesterday, so I guess that trouble has been quietly 
resolved.

That should make this Git patch unnecessary.  (You’re of course still 
welcome to take it if you think build reproducibility with old AsciiDoc 
versions is worthwhile.)

Anders

Re: [PATCH 1/3] Documentation: allow overriding timestamps of generated asciidoc

[Junio C Hamano]

Jonathan Nieder  writes:

>> touch -d @SECONDS isn't POSIX compliant, and non-Linux systems don't
>> provide it.  POSIX only allows certain fixed format, and I assume that
>> non-Linux parties (maybe OpenBSD) will want to have reproducible builds
>> as well.
>
> Interesting.  My knee-jerk preference is still to go with this patch
> as-is for now, since the non-portability only triggers when
> SOURCE_DATE_EPOCH is set.

As long as this patch is kept Debian-only, that is a sensible stance
to take.  I am not sure (note: this is different from "I do not think")
if it is also OK for the wider public, though.

I wondered if this affects the dirtyness of the build, regardless of
how file timestamps are mucked with.  It turns out that we do not
use "describe --dirty" in the GIT-VERSION-GEN script, so perhaps
it would be OK.

Re: [PATCH 1/3] Documentation: allow overriding timestamps of generated asciidoc

[Jonathan Nieder]

Hi,

brian m. carlson wrote:
> On Tue, Nov 21, 2017 at 03:34:32PM -0800, Jonathan Nieder wrote:

>> --- a/Documentation/Makefile
>> +++ b/Documentation/Makefile
>> @@ -410,6 +410,7 @@ $(patsubst %.txt,%.texi,$(MAN_TXT)): %.texi : %.xml
>>  howto-index.txt: howto-index.sh $(wildcard howto/*.txt)
>>  $(QUIET_GEN)$(RM) $@+ $@ && \
>>  '$(SHELL_PATH_SQ)' ./howto-index.sh $(sort $(wildcard howto/*.txt)) 
>> >$@+ && \
>> +$(if $(SOURCE_DATE_EPOCH),touch -d '@$(SOURCE_DATE_EPOCH)' $@+ &&) \
>
> touch -d @SECONDS isn't POSIX compliant, and non-Linux systems don't
> provide it.  POSIX only allows certain fixed format, and I assume that
> non-Linux parties (maybe OpenBSD) will want to have reproducible builds
> as well.

Interesting.  My knee-jerk preference is still to go with this patch
as-is for now, since the non-portability only triggers when
SOURCE_DATE_EPOCH is set.

> It's unfortunate for shell users that this variable is in seconds from
> the epoch, since there's no portable way to format such a time in shell.
> (POSIX doesn't allow date(1) to format anything but the current time.)
>
> My proposed solution was to use Perl to do so, and simply require that
> if you wanted a reproducible build, then you had to have Perl.  That
> would, of course, require a separate variable in the Makefile holding
> the formatted date.
>
> Maybe something like the following in the Makefile:
>
> ifndef NO_PERL
> SOURCE_DATE_TIMESTAMP=$(shell perl -MPOSIX -e 'print strftime("%FT%TZ", 
> gmtime($ENV{SOURCE_DATE_EPOCH}));')
> endif
>
> and then:
>
> + $(if $(SOURCE_DATE_TIMESTAMP),touch -d '$(SOURCE_DATE_TIMESTAMP)' $@+ 
> &&) \

Neat.  I can play with this a little.

http://asciidoc.org/CHANGELOG.html is stale but asciidoc still seems
to be getting changes at https://github.com/asciidoc/asciidoc.  I
wonder how difficult it would be to add any required SOURCE_DATE_EPOCH
support there.

Longer term, I wonder what it would take to move to a markup language
that is more widely known, like commonmark.

Thanks,
Jonathan

Re: [PATCH 1/3] Documentation: allow overriding timestamps of generated asciidoc

[brian m. carlson]

On Tue, Nov 21, 2017 at 03:34:32PM -0800, Jonathan Nieder wrote:
> From: Anders Kaseorg 
> Date: Wed, 30 Nov 2016 22:21:15 -0500
> 
> Allow overriding the timestamp in generated documentation by setting
> SOURCE_DATE_EPOCH to the number of seconds since 1970-01-01 00:00:00
> UTC to use.
> 
> This makes the generated documentation reproducible from the source
> code as long as that variable is set, without losing the last-modified
> dates in the default build.

Thanks for this.  I had planned on either submitting this patch myself,
or working on a similar one, but I ran into the issue I'll mention
below and hadn't finished looking at it.

My research on this determined that Asciidoctor 1.5.5 and newer handle
this properly, because they honor SOURCE_BUILD_EPOCH.  It's only with
AsciiDoc that this is an issue.

> diff --git a/Documentation/Makefile b/Documentation/Makefile
> index 2ab65561af..dfec29f36f 100644
> --- a/Documentation/Makefile
> +++ b/Documentation/Makefile
> @@ -410,6 +410,7 @@ $(patsubst %.txt,%.texi,$(MAN_TXT)): %.texi : %.xml
>  howto-index.txt: howto-index.sh $(wildcard howto/*.txt)
>   $(QUIET_GEN)$(RM) $@+ $@ && \
>   '$(SHELL_PATH_SQ)' ./howto-index.sh $(sort $(wildcard howto/*.txt)) 
> >$@+ && \
> + $(if $(SOURCE_DATE_EPOCH),touch -d '@$(SOURCE_DATE_EPOCH)' $@+ &&) \

touch -d @SECONDS isn't POSIX compliant, and non-Linux systems don't
provide it.  POSIX only allows certain fixed format, and I assume that
non-Linux parties (maybe OpenBSD) will want to have reproducible builds
as well.

It's unfortunate for shell users that this variable is in seconds from
the epoch, since there's no portable way to format such a time in shell.
(POSIX doesn't allow date(1) to format anything but the current time.)

My proposed solution was to use Perl to do so, and simply require that
if you wanted a reproducible build, then you had to have Perl.  That
would, of course, require a separate variable in the Makefile holding
the formatted date.

Maybe something like the following in the Makefile:

ifndef NO_PERL
SOURCE_DATE_TIMESTAMP=$(shell perl -MPOSIX -e 'print strftime("%FT%TZ", 
gmtime($ENV{SOURCE_DATE_EPOCH}));')
endif

and then:

+   $(if $(SOURCE_DATE_TIMESTAMP),touch -d '$(SOURCE_DATE_TIMESTAMP)' $@+ 
&&) \
-- 
brian m. carlson / brian with sandals: Houston, Texas, US
https://www.crustytoothpaste.net/~bmc | My opinion only
OpenPGP: https://keybase.io/bk2204


signature.asc
Description: PGP signature

[PATCH 1/3] Documentation: allow overriding timestamps of generated asciidoc

[Jonathan Nieder]

From: Anders Kaseorg 
Date: Wed, 30 Nov 2016 22:21:15 -0500

Allow overriding the timestamp in generated documentation by setting
SOURCE_DATE_EPOCH to the number of seconds since 1970-01-01 00:00:00
UTC to use.

This makes the generated documentation reproducible from the source
code as long as that variable is set, without losing the last-modified
dates in the default build.

With this change, the package passes Debian's build reproducibility
test (https://wiki.debian.org/ReproducibleBuilds/TimestampsProposal).

The goal is to make it easier to verify that binary packages of open
source projects were built from the source they were claimed to have
been built from.  https://reproducible-builds.org/ has more details.

Signed-off-by: Anders Kaseorg 
Signed-off-by: Jonathan Nieder 
---
Perhaps this should e.g. be taking the latest timestamp of all its
inputs.  That would be straightforward to do, but what's here is what
we've been running with for the past year, so I'd rather stick to it,
at least as a starting point.

Another tweak I'd be interested in is allowing asciidoc to take the
timestamp as a parameter instead of inferring it from mtimes.
Asciidoc accepts an "--attribute footer-style=none" parameter, but I'm
not aware of an "--attribute footer-date=" parameter to keep the
footer but override its date.

 Documentation/Makefile   | 7 +--
 Documentation/technical/api-index.sh | 5 +
 2 files changed, 10 insertions(+), 2 deletions(-)

diff --git a/Documentation/Makefile b/Documentation/Makefile
index 2ab65561af..dfec29f36f 100644
--- a/Documentation/Makefile
+++ b/Documentation/Makefile
@@ -410,6 +410,7 @@ $(patsubst %.txt,%.texi,$(MAN_TXT)): %.texi : %.xml
 howto-index.txt: howto-index.sh $(wildcard howto/*.txt)
$(QUIET_GEN)$(RM) $@+ $@ && \
'$(SHELL_PATH_SQ)' ./howto-index.sh $(sort $(wildcard howto/*.txt)) 
>$@+ && \
+   $(if $(SOURCE_DATE_EPOCH),touch -d '@$(SOURCE_DATE_EPOCH)' $@+ &&) \
mv $@+ $@
 
 $(patsubst %,%.html,$(ARTICLES)) : %.html : %.txt
@@ -420,8 +421,10 @@ WEBDOC_DEST = /pub/software/scm/git/docs
 howto/%.html: ASCIIDOC_EXTRA += -a git-relative-html-prefix=../
 $(patsubst %.txt,%.html,$(wildcard howto/*.txt)): %.html : %.txt
$(QUIET_ASCIIDOC)$(RM) $@+ $@ && \
-   sed -e '1,/^$$/d' $< | \
-   $(TXT_TO_HTML) - >$@+ && \
+   sed -e '1,/^$$/d' $< > $<+ && \
+   $(if $(SOURCE_DATE_EPOCH),touch -d '@$(SOURCE_DATE_EPOCH)' $<+ &&) \
+   $(TXT_TO_HTML) -o $@+ $<+ && \
+   rm $<+ && \
mv $@+ $@
 
 install-webdoc : html
diff --git a/Documentation/technical/api-index.sh 
b/Documentation/technical/api-index.sh
index 9c3f4131b8..07b3909627 100755
--- a/Documentation/technical/api-index.sh
+++ b/Documentation/technical/api-index.sh
@@ -20,6 +20,11 @@
sed -n -e '/^\/\/ table of contents end/,$p' "$skel"
 ) >api-index.txt+
 
+if test "${SOURCE_DATE_EPOCH:+set}"
+then
+   touch -d "@$SOURCE_DATE_EPOCH" api-index.txt+
+fi
+
 if test -f api-index.txt && cmp api-index.txt api-index.txt+ >/dev/null
 then
rm -f api-index.txt+
-- 
2.15.0.448.gf294e3d99a