(Mostly general info, everyone please read) Hi Antonin,
Sorry, somehow I missed this earlier! This work should help with that issue. I’m finding it really great for stuff I do, it’s also really easy to adapt to create a website PR for integrating lots of branch changes, cf. https://github.com/apache/camel-website/pull/729 <https://github.com/apache/camel-website/pull/729>. I’m going to be working on my “referential integrity” idea next (unless more documentation fires break out :-). This will touch every active branch of every subproject, and unless someone objects I’m going to install the local-build stuff everywhere to make it easier for myself :-), so it should soon be pretty easy for everyone to try it on their favorite parts of camel. It’s not a problem for me since I basically only work on the website, but I would imagine two major annoyances of the current system for everyone else are: (1) You have to build the full site at least once, which takes annoyingly long (7 minutes?). (2) You have to build the UI locally, which is tricky on non-linux systems. For (1), building the site manifest as part of the regular website build so the latest can always be downloaded would make it so you can always do a quick local partial build, although you won’t get a very usable local website: just the local stuff will be there locally. It would detect the same errors as a the current local build, but not xref problems into the local changes: this (currently) requires a full build. For (2), one simple option is just checking the built UI bundle into git. A further annoyance that I experience is that the built website is checked into a branch of camel-website. This means whenever you pull camel-website, you get the latest published website which is a lot of changes, along with the tiny changes that might or might not have occurred since your last update. I would like us to move the published website to a different repo. This would also allow us to keep the website history, which would partly solve the problem of docs for outdated component versions that we’ve dropped from the active website. I also think from some comments about the old svn site publishing system that infra expects website history to be preserved, although I haven’t asked explicitly. I’ve set this up in a couple other projects, and it’s pretty easy to do. David Jencks > On Dec 17, 2021, at 1:44 AM, Antonin Stefanutti > <anto...@stefanutti.fr.INVALID> wrote: > > Hi David, > > Thanks a lot for your initiative. I hesitated to comment, both because I'm > definitely foreign to the doc building process, and also as you did not > mention Camel K. However, this, if I understand it correctly would help > solving the following issue: > > https://github.com/apache/camel-k/issues/2278 > <https://github.com/apache/camel-k/issues/2278> > > I think this is critical to have a way to validate doc updates made to > sub-projects before merging so as to make sure they do not fail the main > website build. > > As I said, I'm not familiar with the doc build process, but I'd be happy to > help you on this if there is anything I can do. > > On 17 Dec 2021, at 00:46, David Jencks <david.a.jen...@gmail.com > <mailto:david.a.jen...@gmail.com><mailto:david.a.jen...@gmail.com > <mailto:david.a.jen...@gmail.com>>> wrote: > > Although there have been some comments on the associated PRs, and I’ve merged > some of them, I’m slightly disappointed that there has been no discussion > here, although there have been complaints about one of the problems this is > intended to help with (broken website builds). > > I mention in future work (3) that it might be possible to adapt some of this > for docs PR builds. Here’s what I have in mind: > > 1. Something in the CI detects that there’s a change impacting the docs (or > we just run this for every PR). > > 2. CI for the PR extracts the main repo, the PR repo, the original branch, > and the PR branch and sends them (how?) to camel-website CI. > > 3. CI for camel-website takes this information and constructs a snippet like > this: > > ``` > - require: '@djencks/antora-source-map' > source-map: > - url: ‘$ORIGINAL_REPO' > mapped-url: ‘$PR_REPO' > branches: > - branch: $ORIGINAL_BRANCH > mapped-branch: $PR_BRANCH > ``` > > This gets appended to the antora playbook and the antora build runs. (or a > full build) > > To make it quicker, if a current site-manifest is available, we could also > append something for @djencks/antora-source-watch and > @djencks/antora-site-manifest to only build the stuff from the changed repo > (although this can often be tricky due to interrelationships between > projects, e.g. camel and camel-spring-boot). > > Conceptually this seems pretty simple, but I don’t know enough about GitHub > actions to know whether it can be done or how to do it. I’ve found the GitHub > actions docs hard to understand. If anyone wants to implement this or > provide advice about how to set up the information transfer and triggers that > would make it much more likely to happen. > > David Jencks > > On Dec 14, 2021, at 6:38 PM, David Jencks <david.a.jen...@gmail.com > <mailto:david.a.jen...@gmail.com><mailto:david.a.jen...@gmail.com > <mailto:david.a.jen...@gmail.com>>> wrote: > > I have a POC for a reliable way to locally build small parts of the > (Antora-built) website, e.g. a subproject. > > Prerequisites: > > 1. camel-website must be cloned next to the subproject: e.g. if the “main” > directory is camel-projects, you’ll have camel-projects/camel-website and > camel-projects/camel-quarkus. > Note: it would be possible to have camel-website under the subproject, but > that would make working on more than one subproject implausible. > > 2. You’re using linux, macOS, or another unix-like OS…. there’s a shell > script. > > What to do: > > 1. Initially build the camel ui. This is easy on linux systems and difficult > on macOS; you have to run yarn to recompile some dependencies. It may be > easier to build in docker. > > 2. In the subproject, initially run `./loca-build.sh full` once to perform a > full Antora build that includes your local subproject state, to generate the > ‘site-manifest’ needed for the partial build. > > 3. After this, run `./local-build.sh` to continuously update the full build > with changes in the subproject. This starts a local web server which is > slightly more capable than looking at the file urls. > This does an initial partial build, and then watches for file changes. The > whole site is viewable. > > Limitations: > > 1. At the moment, xref checks from the subproject to the rest of the site are > validated correctly, but xrefs into the subproject from the rest of the site > are not. I’m hoping to extend the site-manifest to fix this. > > 2. You have to do a full build locally once to construct the site manifest > and the local content from the parts of the site outside the subproject. This > takes about 7 minutes on my laptop. If there are significant changes to the > website you need to redo this local build, and there’s no way to detect when > this is needed. > > 3. You have to build the camel ui once which is difficult on non-linux > systems. This could easily be fixed by checking in the built ui bundle > whenever it changes. > > Possible changes/future work: > > 1. The site-manifest can be built at minimal cost during the official website > build, and served from the website. It’s about 2.5 MB, so I don’t think it’s > a significant drain on resources. Doing just this would eliminate the need > for step (2), building the full site locally, and you’d still have good xref > checks, but with just this step you’d get a site where only the subproject > pages are local, and links out of the subproject go to the main site; links > into the subproject would stay on the main site. I don’t like this much, but > perhaps it’s completely adequate for most site development. > > 2. In addition to (1), the zip-publisher could be added to the build, to zip > up the whole Antora part of the site; this could be downloaded instead of > step (2). This would be pretty easy to set up, but the resulting zip is about > 110MB. I don’t know what infra would think of serving this from the root of > the website. This would completely eliminate the need for step (2) unless > you wanted to do a full local build with your changes, perhaps to check xrefs > into the subproject or to check other unusual interactions. > > 3. Some of this may be adaptable to building PRs that change the site > content, with just a subproject PR and not an additional camel-website PR. > This depends on how much information can be sent from one GitHub action to > another in a different repository. > > Current state: > > I’ve set this up for camel-quarkus main in its current state and > camel-kamelets main showing how I propose to generate the kamelet pages on > the fly with Antora. I can set this up for other subprojects on request. > > How it works: > > There are three Antora extensions, some per-subproject playbook additions, > and a shell script. > > @djencks/antora-source-map uses configuration to modify the playbook > source/branch info, for each configured source/branch replacing the apache > GitHub repo with the local clone and branch. (this is what would be used in > "future (3)’) > > @djencks/antora-site-manifest, which is an adaptation and modification of > something Dan Allen developed, generates the site-manifest in the full build > and imports it in the partial build. > > @djencks/antora-source-watch restricts which files are loaded into the > content catalog (rather than being represented by the site-manifest entry) > and sets up a file watcher to detect changes and a web server to show the > site; on a file change it rebuilds the site. browser-watch refreshes any > pages you may be looking at from the site. > > camel-website has a playbook addition that configures the site manifest > generation on a full build. This could be added to the playbook if we > decided to always generate the site-manifest (future (1)). > There are also a couple of yarn scripts to do full and partial antora builds. > > Each subproject has two playbook additions: > - source-map.yml to configure the source-map extension > - source-watch.yml to configure the source-watch extension > > and a script that concatenates the additions to the main Antora playbook to > result in local playbooks for full and partial builds, and runs the builds. > > Where is it?!?!? > > Camel-website issue: https://github.com/apache/camel-website/issues/720 > <https://github.com/apache/camel-website/issues/720> > <https://github.com/apache/camel-website/issues/720 > <https://github.com/apache/camel-website/issues/720>> > camel-website PR https://github.com/apache/camel-website/pull/721 > <https://github.com/apache/camel-website/pull/721> > <https://github.com/apache/camel-website/pull/721 > <https://github.com/apache/camel-website/pull/721>> > camel-quarkus PR https://github.com/apache/camel-quarkus/pull/3385 > <https://github.com/apache/camel-quarkus/pull/3385> > <https://github.com/apache/camel-quarkus/pull/3385 > <https://github.com/apache/camel-quarkus/pull/3385>> > camel-kamelets PR https://github.com/apache/camel-kamelets/pull/630 > <https://github.com/apache/camel-kamelets/pull/630> > <https://github.com/apache/camel-kamelets/pull/630 > <https://github.com/apache/camel-kamelets/pull/630>> > > I think you can try this out by making sure your clones are next to one > another as in prerequisite (1), pulling down my branches, and following the > instructions. > > Comments? > > David Jencks
