Re: Make documentation builds reproducible
On Thu, Feb 8, 2024 at 9:47 PM Peter Eisentraut wrote: > > On 23.01.24 02:06, Peter Smith wrote: > > This has been working forever, but seems to have broken due to commit > > [1] having an undeclared variable. > > > runtime error: file stylesheet-html-common.xsl line 452 element if > > Variable 'autolink.index.see' has not been declared. > > make: *** [html-stamp] Error 10 > > I have committed a fix for this. I have successfully tested docbook-xsl > 1.77.1 through 1.79.*. > Yes, the latest is working for me now. Thanks! == Kind Regards, Peter Smith. Fujitsu Australia.
Re: Make documentation builds reproducible
On 23.01.24 02:06, Peter Smith wrote: This has been working forever, but seems to have broken due to commit [1] having an undeclared variable. runtime error: file stylesheet-html-common.xsl line 452 element if Variable 'autolink.index.see' has not been declared. make: *** [html-stamp] Error 10 I have committed a fix for this. I have successfully tested docbook-xsl 1.77.1 through 1.79.*.
Re: Make documentation builds reproducible
Peter Smith writes: >> IIUC these releases notes [1] say autolink.index.see existed since >> v1.79.1, but unfortunately, that is more recent than my ancient >> installed v1.78.1 > Is anything going to be changed for this? I assume Peter E. is going to address it, but FOSDEM is this week and so a lot of people are going to be busy traveling and conferencing rather than hacking. Things might not happen right away. You could possibly help move things along if you can propose a workable patch. regards, tom lane
Re: Make documentation builds reproducible
On Thu, Jan 25, 2024 at 9:12 AM Peter Smith wrote: > > On Tue, Jan 23, 2024 at 12:32 PM Peter Smith wrote: > > > > On Tue, Jan 23, 2024 at 12:13 PM Tom Lane wrote: > > > > > > Peter Smith writes: > > > > I usually the HTML documentation locally using command: > > > > make STYLE=website html > > > > This has been working forever, but seems to have broken due to commit > > > > [1] having an undeclared variable. > > > > > > Interestingly, that still works fine for me, on RHEL8 with > > > > > > docbook-dtds-1.0-69.el8.noarch > > > docbook-style-xsl-1.79.2-9.el8.noarch > > > docbook-style-dsssl-1.79-25.el8.noarch > > > > > > What docbook version are you using? > > > > > > > [postgres@CentOS7-x64 sgml]$ sudo yum list installed | grep docbook > > docbook-dtds.noarch 1.0-60.el7 @anaconda > > docbook-style-dsssl.noarch1.79-18.el7@base > > docbook-style-xsl.noarch 1.78.1-3.el7 @anaconda > > > > IIUC these releases notes [1] say autolink.index.see existed since > v1.79.1, but unfortunately, that is more recent than my ancient > installed v1.78.1 > > From the release notes: > -- > Robert Stayton: autolink.index.see.xml > > New param to control automatic links in index from see and > seealso to indexterm primary. > -- > > == > [1] https://docbook.sourceforge.net/release/xsl/1.79.1/RELEASE-NOTES.html > Is anything going to be changed for this? Since the recent commit [1] when building the docs now each time I need to first hack (e.g. either the Makefile or stylesheet-html-common.xml) to declare the missing ‘autolink.index.see’ variable. I know that my old OS is approaching EOL but I thought my docbook installation was still valid. == [1] https://github.com/postgres/postgres/commit/b0f0a9432d0b6f53634a96715f2666f6d4ea25a1 Kind Regards, Peter Smith. Fujitsu Australia
Re: Make documentation builds reproducible
On Tue, Jan 23, 2024 at 12:32 PM Peter Smith wrote: > > On Tue, Jan 23, 2024 at 12:13 PM Tom Lane wrote: > > > > Peter Smith writes: > > > I usually the HTML documentation locally using command: > > > make STYLE=website html > > > This has been working forever, but seems to have broken due to commit > > > [1] having an undeclared variable. > > > > Interestingly, that still works fine for me, on RHEL8 with > > > > docbook-dtds-1.0-69.el8.noarch > > docbook-style-xsl-1.79.2-9.el8.noarch > > docbook-style-dsssl-1.79-25.el8.noarch > > > > What docbook version are you using? > > > > [postgres@CentOS7-x64 sgml]$ sudo yum list installed | grep docbook > docbook-dtds.noarch 1.0-60.el7 @anaconda > docbook-style-dsssl.noarch1.79-18.el7@base > docbook-style-xsl.noarch 1.78.1-3.el7 @anaconda > IIUC these releases notes [1] say autolink.index.see existed since v1.79.1, but unfortunately, that is more recent than my ancient installed v1.78.1 >From the release notes: -- Robert Stayton: autolink.index.see.xml New param to control automatic links in index from see and seealso to indexterm primary. -- == [1] https://docbook.sourceforge.net/release/xsl/1.79.1/RELEASE-NOTES.html Kind Regards, Peter Smith. Fujitsu Australia
Re: Make documentation builds reproducible
On Tue, Jan 23, 2024 at 12:13 PM Tom Lane wrote: > > Peter Smith writes: > > I usually the HTML documentation locally using command: > > make STYLE=website html > > This has been working forever, but seems to have broken due to commit > > [1] having an undeclared variable. > > Interestingly, that still works fine for me, on RHEL8 with > > docbook-dtds-1.0-69.el8.noarch > docbook-style-xsl-1.79.2-9.el8.noarch > docbook-style-dsssl-1.79-25.el8.noarch > > What docbook version are you using? > [postgres@CentOS7-x64 sgml]$ sudo yum list installed | grep docbook docbook-dtds.noarch 1.0-60.el7 @anaconda docbook-style-dsssl.noarch1.79-18.el7@base docbook-style-xsl.noarch 1.78.1-3.el7 @anaconda == Kind Regards, Peter Smith. Fujitsu Australia
Re: Make documentation builds reproducible
Peter Smith writes: > I usually the HTML documentation locally using command: > make STYLE=website html > This has been working forever, but seems to have broken due to commit > [1] having an undeclared variable. Interestingly, that still works fine for me, on RHEL8 with docbook-dtds-1.0-69.el8.noarch docbook-style-xsl-1.79.2-9.el8.noarch docbook-style-dsssl-1.79-25.el8.noarch What docbook version are you using? regards, tom lane
Re: Make documentation builds reproducible
Hi,
I usually the HTML documentation locally using command:
make STYLE=website html
~
This has been working forever, but seems to have broken due to commit
[1] having an undeclared variable.
e.g.
[postgres@CentOS7-x64 sgml]$ make STYLE=website html
{ \
echo ""; \
echo ""; \
} > version.sgml
'/usr/bin/perl' ./mk_feature_tables.pl YES
../../../src/backend/catalog/sql_feature_packages.txt
../../../src/backend/catalog/sql_features.txt >
features-supported.sgml
'/usr/bin/perl' ./mk_feature_tables.pl NO
../../../src/backend/catalog/sql_feature_packages.txt
../../../src/backend/catalog/sql_features.txt >
features-unsupported.sgml
'/usr/bin/perl' ./generate-errcodes-table.pl
../../../src/backend/utils/errcodes.txt > errcodes-table.sgml
'/usr/bin/perl' ./generate-keywords-table.pl . > keywords-table.sgml
'/usr/bin/perl' ./generate-targets-meson.pl targets-meson.txt
generate-targets-meson.pl > targets-meson.sgml
'/usr/bin/perl'
../../../src/backend/utils/activity/generate-wait_event_types.pl
--docs ../../../src/backend/utils/activity/wait_event_names.txt
/usr/bin/xmllint --nonet --path . --path . --output postgres-full.xml
--noent --valid postgres.sgml
/usr/bin/xsltproc --nonet --path . --path . --stringparam pg.version
'17devel' --param website.stylesheet 1 stylesheet.xsl
postgres-full.xml
runtime error: file stylesheet-html-common.xsl line 452 element if
Variable 'autolink.index.see' has not been declared.
make: *** [html-stamp] Error 10
==
[1]
https://github.com/postgres/postgres/commit/b0f0a9432d0b6f53634a96715f2666f6d4ea25a1
Kind Regards,
Peter Smith.
Fujitsu Australia
Re: Make documentation builds reproducible
On 20.01.24 23:59, Tom Lane wrote: Peter Eisentraut writes: On 20.01.24 17:03, Tom Lane wrote: * I gather that the point here is to change some generated anchor tags. Would any of these tags be things people would be likely to have bookmarked? No, because the problem is that the anchor names are randomly generated in each build. D'oh. No objection then. Thanks, committed.
Re: Make documentation builds reproducible
Peter Eisentraut writes: > On 20.01.24 17:03, Tom Lane wrote: >> * I gather that the point here is to change some generated anchor >> tags. Would any of these tags be things people would be likely >> to have bookmarked? > No, because the problem is that the anchor names are randomly generated > in each build. D'oh. No objection then. regards, tom lane
Re: Make documentation builds reproducible
On 20.01.24 17:03, Tom Lane wrote: Peter Eisentraut writes: I think there was general agreement with what this patch is doing, but I guess it's too boring to actually review the patch in detail. Let's say, if there are no objections, I'll go ahead and commit it. I re-read the thread and have two thoughts: * We worried about whether this change would be compatible with a (presently unreleased) version of docbook that contains the upstreamed fix. It seems unlikely that there's a problem, but maybe worth checking? The code in the patch is the same code as upstream, so it would behave the same as a new release. * I gather that the point here is to change some generated anchor tags. Would any of these tags be things people would be likely to have bookmarked? No, because the problem is that the anchor names are randomly generated in each build.
Re: Make documentation builds reproducible
Peter Eisentraut writes: > I think there was general agreement with what this patch is doing, but I > guess it's too boring to actually review the patch in detail. Let's > say, if there are no objections, I'll go ahead and commit it. I re-read the thread and have two thoughts: * We worried about whether this change would be compatible with a (presently unreleased) version of docbook that contains the upstreamed fix. It seems unlikely that there's a problem, but maybe worth checking? * I gather that the point here is to change some generated anchor tags. Would any of these tags be things people would be likely to have bookmarked? regards, tom lane
Re: Make documentation builds reproducible
On 20.01.24 03:33, vignesh C wrote: On Fri, 25 Aug 2023 at 01:23, Tristan Partin wrote: On Thu Aug 24, 2023 at 2:30 PM CDT, Tom Lane wrote: "Tristan Partin" writes: On Wed Aug 23, 2023 at 2:24 PM CDT, Peter Eisentraut wrote: Somewhere at PGCon, I forgot exactly where, maybe in the same meeting where we talked about getting rid of distprep, we talked about that the documentation builds are not reproducible (in the sense of https://reproducible-builds.org/). This is easily fixable, Is there anything I am missing? Is Postgres relying on releases older than snapshot-2018-12-07-01? If so, is it possible to up the minimum version? AFAICT the "latest stable release" of docbook-xsl is still 1.79.2, which seems to have been released in 2017, so it's unsurprising that it's missing this fix. It's kind of hard to argue that developers (much less distro packagers) should install unsupported snapshot releases in order to build our docs. Having said that, maybe we should check whether this patch is compatible with those snapshot releases, just in case somebody is using one. I agree with you. Thanks for the pointer. I'm seeing that there has been no activity in this thread for nearly 5 months, I'm planning to close this in the current commitfest unless someone is planning to take it forward. I think there was general agreement with what this patch is doing, but I guess it's too boring to actually review the patch in detail. Let's say, if there are no objections, I'll go ahead and commit it.
Re: Make documentation builds reproducible
On Fri, 25 Aug 2023 at 01:23, Tristan Partin wrote: > > On Thu Aug 24, 2023 at 2:30 PM CDT, Tom Lane wrote: > > "Tristan Partin" writes: > > > On Wed Aug 23, 2023 at 2:24 PM CDT, Peter Eisentraut wrote: > > >> Somewhere at PGCon, I forgot exactly where, maybe in the same meeting > > >> where we talked about getting rid of distprep, we talked about that the > > >> documentation builds are not reproducible (in the sense of > > >> https://reproducible-builds.org/). This is easily fixable, > > > > > Is there anything I am missing? Is Postgres relying on releases older > > > than snapshot-2018-12-07-01? If so, is it possible to up the minimum > > > version? > > > > AFAICT the "latest stable release" of docbook-xsl is still 1.79.2, > > which seems to have been released in 2017, so it's unsurprising that > > it's missing this fix. > > > > It's kind of hard to argue that developers (much less distro packagers) > > should install unsupported snapshot releases in order to build our docs. > > Having said that, maybe we should check whether this patch is compatible > > with those snapshot releases, just in case somebody is using one. > > I agree with you. Thanks for the pointer. I'm seeing that there has been no activity in this thread for nearly 5 months, I'm planning to close this in the current commitfest unless someone is planning to take it forward. Regards, Vignesh
Re: Make documentation builds reproducible
On Thu Aug 24, 2023 at 2:30 PM CDT, Tom Lane wrote: "Tristan Partin" writes: > On Wed Aug 23, 2023 at 2:24 PM CDT, Peter Eisentraut wrote: >> Somewhere at PGCon, I forgot exactly where, maybe in the same meeting >> where we talked about getting rid of distprep, we talked about that the >> documentation builds are not reproducible (in the sense of >> https://reproducible-builds.org/). This is easily fixable, > Is there anything I am missing? Is Postgres relying on releases older > than snapshot-2018-12-07-01? If so, is it possible to up the minimum > version? AFAICT the "latest stable release" of docbook-xsl is still 1.79.2, which seems to have been released in 2017, so it's unsurprising that it's missing this fix. It's kind of hard to argue that developers (much less distro packagers) should install unsupported snapshot releases in order to build our docs. Having said that, maybe we should check whether this patch is compatible with those snapshot releases, just in case somebody is using one. I agree with you. Thanks for the pointer. -- Tristan Partin Neon (https://neon.tech)
Re: Make documentation builds reproducible
"Tristan Partin" writes: > On Wed Aug 23, 2023 at 2:24 PM CDT, Peter Eisentraut wrote: >> Somewhere at PGCon, I forgot exactly where, maybe in the same meeting >> where we talked about getting rid of distprep, we talked about that the >> documentation builds are not reproducible (in the sense of >> https://reproducible-builds.org/). This is easily fixable, > Is there anything I am missing? Is Postgres relying on releases older > than snapshot-2018-12-07-01? If so, is it possible to up the minimum > version? AFAICT the "latest stable release" of docbook-xsl is still 1.79.2, which seems to have been released in 2017, so it's unsurprising that it's missing this fix. It's kind of hard to argue that developers (much less distro packagers) should install unsupported snapshot releases in order to build our docs. Having said that, maybe we should check whether this patch is compatible with those snapshot releases, just in case somebody is using one. regards, tom lane
Re: Make documentation builds reproducible
On Wed Aug 23, 2023 at 2:24 PM CDT, Peter Eisentraut wrote: Somewhere at PGCon, I forgot exactly where, maybe in the same meeting where we talked about getting rid of distprep, we talked about that the documentation builds are not reproducible (in the sense of https://reproducible-builds.org/). This is easily fixable, the fix is available upstream (https://github.com/docbook/xslt10-stylesheets/issues/54) but not released. We can backpatch that into our customization layer. The attached patch shows it. I am a tiny bit confused here. The commit that solved the issue was merged into the master branch in 2018. GitHub lists the lastest release as being in 2020. A quick git command shows this has been in releases since December of 2018. $ git --no-pager tag --contains 0763160 ndw-test-001 snapshot-2018-12-07-01 snapshot-ndw-test/2019-10-04 snapshot/2018-09-28-172 snapshot/2018-09-28-173 snapshot/2018-09-28-174 snapshot/2018-09-28-175 snapshot/2018-09-29-176 snapshot/2018-09-29-177 snapshot/2018-09-30-178 snapshot/2018-09-30-179 snapshot/2018-10-01-180 snapshot/2018-10-02-183 snapshot/2018-10-02-184 snapshot/2018-10-16-185 snapshot/2018-10-16-186 snapshot/2018-10-21-188 snapshot/2018-11-01-191 snapshot/2019-10-05-bobs snapshot/2020-05-28-pdesjardins snapshot/2020-06-03 Is there anything I am missing? Is Postgres relying on releases older than snapshot-2018-12-07-01? If so, is it possible to up the minimum version? I had actually often wanted this during development. When making documentation tooling changes, it's useful to be able to compare the output before and after, and this will eliminate false positives in that. This patch addresses both the HTML and the FO output. The man output is already reproducible AFAICT. Note that the final PDF output is currently not reproducible; that's a different issue that needs to be fixed in FOP. (See https://wiki.debian.org/ReproducibleBuilds/TimestampsInPDFGeneratedByApacheFOP.) I think reproducibility is very important. Thanks for taking this on! -- Tristan Partin Neon (https://neon.tech)
Make documentation builds reproducible
Somewhere at PGCon, I forgot exactly where, maybe in the same meeting where we talked about getting rid of distprep, we talked about that the documentation builds are not reproducible (in the sense of https://reproducible-builds.org/). This is easily fixable, the fix is available upstream (https://github.com/docbook/xslt10-stylesheets/issues/54) but not released. We can backpatch that into our customization layer. The attached patch shows it. I had actually often wanted this during development. When making documentation tooling changes, it's useful to be able to compare the output before and after, and this will eliminate false positives in that. This patch addresses both the HTML and the FO output. The man output is already reproducible AFAICT. Note that the final PDF output is currently not reproducible; that's a different issue that needs to be fixed in FOP. (See https://wiki.debian.org/ReproducibleBuilds/TimestampsInPDFGeneratedByApacheFOP.)From 9f2c7262445b6868c1d2c9f2d75ac1153b7c2a64 Mon Sep 17 00:00:00 2001 From: Peter Eisentraut Date: Wed, 23 Aug 2023 16:38:44 +0200 Subject: [PATCH] Make documentation builds reproducible --- doc/src/sgml/stylesheet-fo.xsl | 259 doc/src/sgml/stylesheet-html-common.xsl | 234 + 2 files changed, 493 insertions(+) diff --git a/doc/src/sgml/stylesheet-fo.xsl b/doc/src/sgml/stylesheet-fo.xsl index 5e7e132480..aff717ddbc 100644 --- a/doc/src/sgml/stylesheet-fo.xsl +++ b/doc/src/sgml/stylesheet-fo.xsl @@ -1,4 +1,8 @@ +http://docbook.sourceforge.net/release/xsl/current/common/entities.ent";> +%common.entities; +]> http://www.w3.org/1999/XSL/Transform"; version="1.0" xmlns:fo="http://www.w3.org/1999/XSL/Format";> @@ -138,4 +142,259 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + http://www.renderx.com/XSL/Extensions"; xmlns:axf="http://www.antennahouse.com/names/XSL/Extensions";> + + + +ientry- + + + + + + true + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + ientry- + + + + + + + + + ( + + + + + + + + + + + + + + + + + + + + + + + + +) + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +ientry- + + + + + + + + + ( + + + + + + + + + + + + + + + + + + + + + + + + + ) + + + + + + + diff --git a/doc/src/sgml/stylesheet-html-common.xsl b/doc/src/sgml/stylesheet-html-common.xsl index a368e0e199..7f541c0988 100644 --- a/doc/src/sgml/stylesheet-html-common.xsl +++ b/doc/src/sgml/stylesheet-html-common.xsl @@ -436,4 +436,238 @@ set toc,title + + + + + + +
