Re: [PATCH 0/4] v2 of Option handling: add documentation URLs

[Mark Wielaard]

Hi David,

On Fri, Dec 08, 2023 at 06:35:56PM -0500, David Malcolm wrote:
> On Tue, 2023-11-21 at 23:43 +, Joseph Myers wrote:
> > On Tue, 21 Nov 2023, Tobias Burnus wrote:
> > 
> > > On 21.11.23 14:57, David Malcolm wrote:
> > > > On Tue, 2023-11-21 at 02:09 +0100, Hans-Peter Nilsson wrote:
> > > > > Sorry for barging in though I did try finding the relevant
> > > > > discussion, but is committing this generated stuff necessary?
> > > > > Is it because we don't want to depend on Python being
> > > > > present at build time?
> > > > Partly, yes, [...]
> > > 
> > > I wonder how to ensure that this remains up to date. Should there
> > > be an item at
> > > 
> > > https://gcc.gnu.org/branching.html and/or
> > > https://gcc.gnu.org/releasing.html similar to the .pot generation?
> >
> > My suggestion earlier in the discussion was that it should be
> > added to the post-commit CI discussed starting at
> >  (I
> > think that CI is now in operation).  These are generated files
> > that ought to be kept up to date with each commit that affects
> > .opt files, unlike the .pot files where the expectation is that
> > they should be up to date for releases and updated from time to
> > time at other times for submission to the TP.
> > 
> I had a go at scripting the testing of this, but I am terrible at shell
> scripts (maybe I should use Python?).  Here's what I have so far:
> 
> $ cat contrib/regenerate-index-urls.sh
> 
> set -x
> 
> SRC_DIR=$1
> BUILD_DIR=$2
> NUM_JOBS=$3
> 
> # FIXME: error-checking!
> 
> mkdir -p $BUILD_DIR || exit 1
> cd $BUILD_DIR
> $SRC_DIR/configure --disable-bootstrap --enable-languages=c,d,fortran || exit 
> 2
> make html-gcc -j$NUM_JOBS || exit 3
> cd gcc || exit 4
> make regenerate-opt-urls || exit 5
> cd $SRC_DIR
> (git diff $1 > /dev/null ) && echo "regenerate-opt-urls needs to be run and 
> the results committed" || exit 6
> 
> # e.g.
> #  time bash contrib/regenerate-index-urls.sh $(pwd) $(pwd)/../build-ci 64
> 
> This takes about 100 seconds of wallclock on my 64-core box (mostly
> configuring gcc, which as well as the usual sequence of unparallelized
> tests seems to require building libiberty and lto-plugin).  Is that
> something we want to do on every commit?  Is implementing the CI a
> blocker for getting the patches in? (if so, I'll likely need some help)

The CI builers don't have 64-cores, but a couple of hundred seconds
shouldn't be an issue to do on each commit (OSUOSL just got us a
second x86_64 container builder for larger jobs). The above can easily
be added to the existing gcc-autoregen builder:
https://builder.sourceware.org/buildbot/#/builders/gcc-autoregen
https://sourceware.org/cgit/builder/tree/builder/master.cfg#n3453

Once your patch is in please feel free to sent an email to
build...@sourceware.org
https://sourceware.org/mailman/listinfo/buildbot
And we'll add the above build steps and update the autotools
Containerfile to include the fortran (gfortran?) and d (gdc?) build
dependencies.

Cheers,

Mark

Re: [PATCH 0/4] v2 of Option handling: add documentation URLs

[David Malcolm]

On Tue, 2023-11-21 at 23:43 +, Joseph Myers wrote:
> On Tue, 21 Nov 2023, Tobias Burnus wrote:
> 
> > On 21.11.23 14:57, David Malcolm wrote:
> > > On Tue, 2023-11-21 at 02:09 +0100, Hans-Peter Nilsson wrote:
> > > > Sorry for barging in though I did try finding the relevant
> > > > discussion, but is committing this generated stuff necessary?
> > > > Is it because we don't want to depend on Python being
> > > > present at build time?
> > > Partly, yes, [...]
> > 
> > I wonder how to ensure that this remains up to date. Should there
> > be an
> > item at
> > 
> > https://gcc.gnu.org/branching.html and/or
> > https://gcc.gnu.org/releasing.html similar to the .pot generation?
> 
> My suggestion earlier in the discussion was that it should be added
> to the 
> post-commit CI discussed starting at 
>  (I
> think 
> that CI is now in operation).  These are generated files that ought
> to be 
> kept up to date with each commit that affects .opt files, unlike the
> .pot 
> files where the expectation is that they should be up to date for
> releases 
> and updated from time to time at other times for submission to the
> TP.

I had a go at scripting the testing of this, but I am terrible at shell
scripts (maybe I should use Python?).  Here's what I have so far:

$ cat contrib/regenerate-index-urls.sh

set -x

SRC_DIR=$1
BUILD_DIR=$2
NUM_JOBS=$3

# FIXME: error-checking!

mkdir -p $BUILD_DIR || exit 1
cd $BUILD_DIR
$SRC_DIR/configure --disable-bootstrap --enable-languages=c,d,fortran || exit 2
make html-gcc -j$NUM_JOBS || exit 3
cd gcc || exit 4
make regenerate-opt-urls || exit 5
cd $SRC_DIR
(git diff $1 > /dev/null ) && echo "regenerate-opt-urls needs to be run and the 
results committed" || exit 6

# e.g.
#  time bash contrib/regenerate-index-urls.sh $(pwd) $(pwd)/../build-ci 64

This takes about 100 seconds of wallclock on my 64-core box (mostly
configuring gcc, which as well as the usual sequence of unparallelized
tests seems to require building libiberty and lto-plugin).  Is that
something we want to do on every commit?  Is implementing the CI a
blocker for getting the patches in? (if so, I'll likely need some help)

As it turned out, I hadn't regenerated the .opt.urls in my working copy
for a couple of weeks, leading to a correct-looking patch containing
things like:

@@ -154,8 +157,8 @@ 
UrlSuffix(gcc/Warning-Options.html#index-Wbuiltin-declaration-mismatch) LangUrlS
 Wbuiltin-macro-redefined
 UrlSuffix(gcc/Warning-Options.html#index-Wbuiltin-macro-redefined)
 
-Wc11-c2x-compat
-UrlSuffix(gcc/Warning-Options.html#index-Wc11-c2x-compat)
+Wc11-c23-compat
+UrlSuffix(gcc/Warning-Options.html#index-Wc11-c23-compat)
 
 Wc90-c99-compat
 UrlSuffix(gcc/Warning-Options.html#index-Wc90-c99-compat)

so I think the idea works; and the only issue for not regenerating was
some missing/out-of-date URLs.

Dave

Re: [PATCH 0/4] v2 of Option handling: add documentation URLs

[Joseph Myers]

On Tue, 21 Nov 2023, Tobias Burnus wrote:

> On 21.11.23 14:57, David Malcolm wrote:
> > On Tue, 2023-11-21 at 02:09 +0100, Hans-Peter Nilsson wrote:
> > > Sorry for barging in though I did try finding the relevant
> > > discussion, but is committing this generated stuff necessary?
> > > Is it because we don't want to depend on Python being
> > > present at build time?
> > Partly, yes, [...]
> 
> I wonder how to ensure that this remains up to date. Should there be an
> item at
> 
> https://gcc.gnu.org/branching.html and/or
> https://gcc.gnu.org/releasing.html similar to the .pot generation?

My suggestion earlier in the discussion was that it should be added to the 
post-commit CI discussed starting at 
 (I think 
that CI is now in operation).  These are generated files that ought to be 
kept up to date with each commit that affects .opt files, unlike the .pot 
files where the expectation is that they should be up to date for releases 
and updated from time to time at other times for submission to the TP.

-- 
Joseph S. Myers
jos...@codesourcery.com

Re: [PATCH 0/4] v2 of Option handling: add documentation URLs

[David Malcolm]

On Tue, 2023-11-21 at 15:12 +0100, Tobias Burnus wrote:
> On 21.11.23 14:57, David Malcolm wrote:
> > On Tue, 2023-11-21 at 02:09 +0100, Hans-Peter Nilsson wrote:
> > > Sorry for barging in though I did try finding the relevant
> > > discussion, but is committing this generated stuff necessary?
> > > Is it because we don't want to depend on Python being
> > > present at build time?
> > Partly, yes, [...]
> 
> I wonder how to ensure that this remains up to date. Should there be
> an
> item at
> 
> https://gcc.gnu.org/branching.html and/or
> https://gcc.gnu.org/releasing.html similar to the .pot generation?

Good point; the releasing.html's Preparations could have as point 6:

  Regenerate the .opt.urls files by running "make regenerate-opt-urls"
in the build/gcc directory.

or similar.

We should also probably recommend that people do that when adding a new
option; is there a documentation page for that?

Dave

Re: [PATCH 0/4] v2 of Option handling: add documentation URLs

[Tobias Burnus]

On 21.11.23 14:57, David Malcolm wrote:

On Tue, 2023-11-21 at 02:09 +0100, Hans-Peter Nilsson wrote:

Sorry for barging in though I did try finding the relevant
discussion, but is committing this generated stuff necessary?
Is it because we don't want to depend on Python being
present at build time?

Partly, yes, [...]


I wonder how to ensure that this remains up to date. Should there be an
item at

https://gcc.gnu.org/branching.html and/or
https://gcc.gnu.org/releasing.html similar to the .pot generation?

Tobias

-
Siemens Electronic Design Automation GmbH; Anschrift: Arnulfstraße 201, 80634 
München; Gesellschaft mit beschränkter Haftung; Geschäftsführer: Thomas 
Heurung, Frank Thürauf; Sitz der Gesellschaft: München; Registergericht 
München, HRB 106955

Re: [PATCH 0/4] v2 of Option handling: add documentation URLs

[David Malcolm]

On Tue, 2023-11-21 at 02:09 +0100, Hans-Peter Nilsson wrote:
> > From: David Malcolm 
> > Date: Thu, 16 Nov 2023 09:28:54 -0500
> 
> > How is this looking for trunk?
> > 
> > Thanks
> > Dave
> > 
> > 
> > David Malcolm (4):
> >   options: add gcc/regenerate-opt-urls.py
> >   Add generated .opt.urls files
> >   opts: add logic to generate options-urls.cc
> >   options: wire up options-urls.cc into gcc_urlifier
> > 
> >  gcc/Makefile.in  |   29 +-
> >  gcc/ada/gcc-interface/lang.opt.urls  |   30 +
> >  gcc/analyzer/analyzer.opt.urls   |  206 ++
> >  gcc/c-family/c.opt.urls  | 1409 ++
> >  gcc/common.opt.urls  | 1832
> > ++
> >  gcc/config/aarch64/aarch64.opt.urls  |   84 +
> >  gcc/config/alpha/alpha.opt.urls  |   76 +
> >  gcc/config/alpha/elf.opt.urls    |    2 +
> >  gcc/config/arc/arc-tables.opt.urls   |    2 +
> 
> [... etc .opt.urls particularly in gcc/config/*
> autogenerated for each *.opt ...]
> 
> Sorry for barging in though I did try finding the relevant
> discussion, but is committing this generated stuff necessary?
> Is it because we don't want to depend on Python being
> present at build time?

Partly, yes, but also because this information is generated from the
generated HTML for the user manuals: gcc, gdc, and gfortran, and it
gets used to generate "optionlist", which is a dependency for all of
the rest of the "gcc" subdirectory.  Doing it automatically would make
the generation of the HTML docs for gcc, d and gfortran be a hard
dependency early on in the build, which seems to me to be a bad idea -
better to have this rarely-changing and non-critical information be
regenerated when it's needed, and not impose the requirement to build
the HTML docs for all these langs on all builds of gcc.

> 
> If nothing else, can you add a few lines to the commit
> message why this can't be/is preferably not done at build
> time?

The wording here:
https://gcc.gnu.org/pipermail/gcc-patches/2023-November/636060.html
may be what you're looking for.

Maybe that text (or something like it) should go in as a big comment at
the top of regenerate-opt-urls.py ?

Hope this clarifies things
Dave

Re: [PATCH 0/4] v2 of Option handling: add documentation URLs

[Hans-Peter Nilsson]

> From: David Malcolm 
> Date: Thu, 16 Nov 2023 09:28:54 -0500

> How is this looking for trunk?
> 
> Thanks
> Dave
> 
> 
> David Malcolm (4):
>   options: add gcc/regenerate-opt-urls.py
>   Add generated .opt.urls files
>   opts: add logic to generate options-urls.cc
>   options: wire up options-urls.cc into gcc_urlifier
> 
>  gcc/Makefile.in  |   29 +-
>  gcc/ada/gcc-interface/lang.opt.urls  |   30 +
>  gcc/analyzer/analyzer.opt.urls   |  206 ++
>  gcc/c-family/c.opt.urls  | 1409 ++
>  gcc/common.opt.urls  | 1832 ++
>  gcc/config/aarch64/aarch64.opt.urls  |   84 +
>  gcc/config/alpha/alpha.opt.urls  |   76 +
>  gcc/config/alpha/elf.opt.urls|2 +
>  gcc/config/arc/arc-tables.opt.urls   |2 +

[... etc .opt.urls particularly in gcc/config/*
autogenerated for each *.opt ...]

Sorry for barging in though I did try finding the relevant
discussion, but is committing this generated stuff necessary?
Is it because we don't want to depend on Python being
present at build time?

If nothing else, can you add a few lines to the commit
message why this can't be/is preferably not done at build
time?

brgds, H-P