This is an automated email from the ASF dual-hosted git repository. mck pushed a commit to branch trunk in repository https://gitbox.apache.org/repos/asf/cassandra-website.git
commit 07f0e129404014c8b1a5921d05aea0d1d1111760 Author: Anthony Grasso <[email protected]> AuthorDate: Sat Jun 26 13:05:29 2021 +1000 CASSANDRA-16066: Update README * Added information and examples on how to use the docker wrapper script. * Added rationale information about why repository structure and tooling used. * Removed information about old build process and commands that have been replaced. * Updated and reused old information that is still applicable to new tooling. patch by Anthony Grasso; reviewed by Mick Semb Wever, Lorina Poland, Melissa Logan, Paul Au for CASSANDRA-16066 --- README.md | 318 ++++++++++++++++++++++++++++++++++++++++++++++---------------- 1 file changed, 236 insertions(+), 82 deletions(-) diff --git a/README.md b/README.md index 8103c1a..fda4bb4 100644 --- a/README.md +++ b/README.md @@ -1,13 +1,31 @@ # Apache Cassandra website -TODO: Add notes about the repository structure +The website repository code is separated into two main parts. These parts are represented by the directories at the root level of the project. Specifically the structure of the repository is: + +``` +ROOT + - site-content + - site-ui +``` + +## Site UI + +The 'site-ui' directory contains only the UI styling files that determines the look and feel of the site. A *ui-bundle.zip* file containing the styling information will be generated from the contents of this directory. Generation of the *ui-bundle.zip* will be done using `gulp` launched inside a Docker container. + +## Site Content + +The 'site-content' directory contains all the raw page information e.g. where to download, developer guidelines, how to commit patches, etc. The live website HTML is generated from the contents of this directory. Generation of the HTML content is done by `antora` launched inside a Docker container. As part of the website HTML generation, the ui-bundle.zip file, and the Cassandra documentation location are passed to `antora`. It uses the ui-bundle.zip to style the website. The Cassandra d [...] + +## Further Reading + +For further details about why the directories are separated as described above and why we use `antora` please see the [Details](#details) section at the bottom of this page. ## Development Cycle Making changes to the website content can be done using the following steps. 1. Preview changes locally. See below for further details about how to do this. -2. Commit changes in `site-content/source` to a fork and branch. +2. Commit changes in `site-content/source/html` to a fork and branch. 3. Create a pull request back to this repository. 3. Get the pull request reviewed and merged to `trunk`. 4. Preview the rendered site on https://cassandra.staged.apache.org/ (wait til [ci-cassandra.apache.org](https://ci-cassandra.apache.org/job/cassandra-website/) has deployed it). @@ -16,7 +34,7 @@ Making changes to the website content can be done using the following steps. # Developer Quickstart -To test changes before committing, it is a requirement that you build the website locally. Building the Apache Cassandra website takes a number of steps. To make things easier we have provided a Docker container which can build the full website in a few simple commands and have it ready to commit via git. +To test changes before committing, it is a requirement that you build the website locally. Building the Apache Cassandra website takes a number of steps. To make things easier we have provided a suite of tools to build the full website in a few simple commands and have it ready to commit via git. Please see the [Details](#details) section at the bottom of this page for further details about the tooling. ## Building Prerequisites @@ -31,150 +49,286 @@ $ git clone https://github.com/apache/cassandra-website.git $ cd ./cassandra-website ``` -To build the website run the following commands from within the `./cassandra-website` directory (assuming you used the above clone command): +A `run.sh` wrapper script has been provided to simplify generating the docs and building the site. It provides a single commandline interface that generates the docker commands to run the website and UI docker containers. + +The script has the following usage format ```bash -$ export BUILD_USER=build -$ docker-compose build website -$ docker-compose run website +$ ./run.sh <COMPONENT> <COMMAND> [OPTIONS] ``` -:warning: *Tip:* In order to prevent root-owned modified files in your repository, the container user, `build`, is set up with a default UID=1000:GID=1000, which is usually the first user configured on a linux machine. If your local user is different you should set up the container user with your local host user's UID:GID, replace the above with: +A complete list of components, commands and options can be found by running the following command. -:warning: *Tip:* Building cassandra-website may fail if the `CLOUDSDK_PYTHON` environment variable is not set on your machine. For example, set the environment variable to export `CLOUDSDK_PYTHON=/usr/bin/python2`. +```bash +$ ./run.sh -h +``` + +To build the website only, run the following command from within the `./cassandra-website` directory (assuming you used the above clone command). ```bash -$ export BUILD_USER=build -$ docker-compose build --build-arg UID_ARG=$(id -u) --build-arg GID_ARG=$(id -g) website -$ docker-compose run website +$ ./run.sh website build ``` -Once building has completed, the site content will be in the `./cassandra-website/site-content/build` directory ready to be tested. +This will build the website content using your local copy of the cassandra-website, and the current checked-out branch. Use this command if you want to make a change to a top-level webpage without building the docs for any versions of cassandra. -### Customising the Website Build for Development +Once building has completed, the HTML content will be in the `./site-content/build/html/` directory ready to be reviewed and committed. -#### You can use your own Cassandra fork +:warning: *Tip:* In order to prevent root-owned modified files in your repository, the container executes operations as a non-root user. By default, the user is `build` and has the user and group permissions set to `UID=1000` and `GID=1000` respectfully. These permissions are usually the first user configured on a linux machine. -The container contains a local copy of the Apache Cassandra repository. The document generation process commits the generated AsciiDoc to the local repository. This is done to allow Antora to render to HTML the versioned documentation and website using the committed AsciiDoc in each branch. +If your local user has different user and group permissions you can set up the container user with your local UID:GID. In addition, you can set the build user in the container your local username. These changes can be made when building the container using the following command: -If you have a Cassandra fork you would like to use instead, you can override the repository used when building the container: +```bash +$ ./run.sh website container -a BUILD_USER_ARG:$(whoami) -a UID_ARG:$(id -u) -a GID_ARG:$(id -g) +``` + +If you need to customise the container user as noted above, you must do this before you build the website or run any other website command. + +## Build the Website when Developing + +The website tooling is very flexible and allows for a wide range of development scenarios. +### Build the website from a different branch + +You can tell the website builder to use a different branch to the one you are on. This can be done using the following command. + +```bash +$ ./run.sh website build -b cassandra-website:my_branch ``` -$ export BUILD_USER=build -$ docker-compose build --build-arg CASSANDRA_REPOSITORY_URL_ARG=<git_url_to_your_fork> website -$ docker-compose run website + +This will build the website content using your local copy of the cassandra-website, and the branch named `my_branch`. + +### Build the website using a local clone of the repository + +You can tell the website builder to use a different clone or fork of the repository. + +To build using another local copy of the cassandra-website run the following command. + +```bash +$ ./run.sh website build -u cassandra-website:/local/path/to/another/clone/of/cassandra-website ``` -Note that you can specify multiple `--build-arg` options when calling `docker-compose build`. So if you wanted to use a different user ID, group ID and repository you could run the following command: +This will build the website using the contents of the local repository located in */local/path/to/another/clone/of/cassandra-website* +### Build the website using a remote clone of the repository + +To build using a remote copy of the cassandra-website run the following command. + +```bash +$ ./run.sh website build -u cassandra-website:https://github.com/my_orgranisation/cassandra-website.git ``` -$ export BUILD_USER=build -$ docker-compose build \ - --build-arg CASSANDRA_REPOSITORY_URL_ARG=<git_url_to_your_fork> \ - --build-arg UID_ARG=$(id -u) \ - --build-arg GID_ARG=$(id -g) \ - website -$ docker-compose run website + +This will build the website using the contents of the remote repository located at *https://github.com/my_orgranisation/cassandra-website.git* + +:warning: *Tip:* The `HEAD` branch of the Cassandra Website repository is always used by default unless an alternative branch is specified. + +In both cases above the `HEAD` branch is used and translates to different branches. In first case where the repository is local, `HEAD` will translate to the currently checked out branch. In the second case where the repository is remote and needs to be cloned, `HEAD` will translate to default branch selected when the repository is cloned. You can specify a different branch using the `-b` option as per the example in [Build a different branch](#build-a-different-branch). + +## Previewing the Website + +An offline preview mode exists if you want to view the website as you make changes to the content. Preview mode can be launched using the following command. + +```bash +$ ./run.sh website preview ``` -In addition to customising the container build, you can customise the environment variables used inside the container. These can be used to influence Antora's rending of the site. For example, Antora will generate documentation for all Cassandra releases starting at 3.11.5. Hence, if you have a specific branch in your fork of Cassandra that you would like to render, you can override the variable the specifies the branches to render. +The site can be viewed on [http://localhost:5151](http://localhost:5151). + +The `preview` command operates the same as the `build` command. It will build the website content using your local copy of the cassandra-website, and the current checked-out branch. Additionally, it will then start a webserver to serve the HTML and a process that monitors the content files in your local copy of the repository. If a change is made to a content file, the website HTML will automatically be regenerated. + +Press `Ctrl+C` to stop the preview server and end the continuous build. + +:warning: *Tip:* You may need to refresh your browser when the auto rendering of the site is complete. + +All options that are available in the `build` command can be used by the `preview` command. Hence, the options used in the previous examples can be specified when using the `preview` command. -TODO: Explain that you need an env.dev file with variables in it and to run `docker-compose run website-dev` +# Developer Advanced Usage Guide -Below is a select list of environment variables that can be overridden to control Antora's rendering of the site and why you would want to override them. +The cassandra-website tooling can also be used to perform a number of other operations: -TODO: Add reason why you would override these -`CASSANDRA_REPOSITORY_URL` -`CASSANDRA_VERSIONS` -`CASSANDRA_WEBSITE_REPOSITORY_URL` -`CASSANDRA_WEBSITE_VERSIONS` -`UI_BUNDLE_ZIP_URL` +* Generate the Apache Cassandra documentation. +* Update the website styling and behaviour. +## Generating the Cassandra Documentation +The website tooling provides the ability to generate the Cassandra AsciiDoc (.adoc) files. The content of the AsciiDoc files are sourced from nodetool and the cassandra.yaml configuration file. The AsciiDoc files need to exist in the Cassandra repository as either commits to the branch or changes ready to be committed before `antora` can generate the HTML formatted equivalent for publication to the Cassandra website. -# WARNING: From here on is all wrong!!!! +By default, the Docker container that generates the version documentation will clone the Apache Cassandra project within the container when generating the documentation. The document generation process commits the generated AsciiDoc to this repository. This is done so `antora` can generate website HTML as well as the HTML for the various versions of the documentation using the committed AsciiDoc in each branch. -### Previewing the Website Build +You can use your own local copy or fork of the Cassandra repository as the source for the versioned documentation. Doing this will also allow you to view and access the generated AsciiDoc files. To use your own copy or fork of the Cassandra repository, you can specify a local path to the `run.sh` script. In this case, when rendering multiple versions of the documentation, any changes to the generated AsciiDoc files will be committed to your local copy or fork. If multiple source branches [...] -The fastest way to preview the site is to run the following: +### Generate documentation using a local clone of the repository + +To generate the latest version of the Cassandra docs using a local copy or fork of the Cassandra repository, run the following command. ```bash -$ docker-compose up preview +$ ./run.sh website docs -u cassandra:/local/path/to/cassandra/repository -b cassandra:trunk ``` -Then view the site on http://localhost:8000 +The output of this command will be AsciiDoc (`.adoc`) files that `antora` can render into HTML. Note that `antora` is never executed when only the Cassandra versioned documentation is generated. + +If you are generating only the Cassandra version documentation, you should always specify a local copy of a Cassandra repository. This is because the generated AsciiDoc files will be placed in the Cassandra directory or committed to the branch used to generate them. You will then be able to view the contents of the AsciiDoc files. + +If you are generating only the Cassandra version documentation, and you specify a remote Cassandra repository location, the generated AsciiDoc files will be inaccessible. This is because the Cassandra repository will be cloned inside the container before the AsciiDoc files are generated. In this case a warning message will be displayed, and you will be asked whether to proceed with the operation. Specifying a remote Cassandra repository location is useful in cases where the HTML website [...] + +### Generate documentation for multiple Cassandra versions -If you want to preview the site as you are editing it run this command: +To generate multiple versions of the Cassandra documentation using a local copy or fork of the Cassandra repository, run the following command. ```bash -$ docker-compose build cassandra-website -$ docker-compose up cassandra-website-serve +$ ./run.sh website docs -u cassandra:/local/path/to/cassandra/repository -b cassandra:trunk,cassandra-3.11,my_branch ``` -For information about the site layout see the **Layout** section of [README](src/README#layout) in the _src_ directory. +In the above command, multiple branches separated by a comma (`,`) can be specified in the `-b` option. +## Generating the website and versioned documentation HTML at the sametime -How ci-cassandra.apache.org continuously deploys to cassandra.staged.apache.org -------------------------------------------------------------------------------- +The website tooling can be used to run a complete end-to-end generation of the HTML website and versioned documentation. That is, it can first generate the AsciiDoc files for each Cassandra version, and then launch `antora` to generate the HTML website and versioned documentation for publication. -The `cassandra-website` job on ci-cassandra.apache.org performs the following steps to CD master builds to cassandra.staged.apache.org +### Generate the website and documentation using pre-generated Cassandra AsciiDoc in local repositories -``` -git checkout asf-staging -git reset --hard origin/master -docker-compose build --build-arg UID=$(id -u) --build-arg GID=$(id -g) cassandra-website -docker-compose run cassandra-website -#git status -git add content/ src/doc/ -git commit -a -m "generate docs for $(git rev-parse --short HEAD)" -#git show --stat HEAD -git push -f origin asf-staging +If you have already generated the Cassandra AsciiDoc (`.adoc`) files and committed them to your repository, you can skip the Cassandra AsciiDoc generation process. + +To build the website using a local clone of the Cassandra repository that contains the generated AsciiDoc files, and the Cassandra Website run the following command. + +```bash +$ ./run.sh \ + website \ + build \ + -i \ + -u cassandra:/local/path/to/cassandra/repository \ + -u cassandra-website:/local/path/to/cassandra-website/repository ``` +In the above command, the `-i` option is used to tell the tooling to include the Cassandra repository when `antora` is generating the HTML. This ensures the versioned documentation HTML is generated along with the website HTML. In this case, exiting AsciiDoc files in the Cassandra repository are used to generate the versioned documentation HTML. That is, no additional operations are run to pre-generate the Cassandra AsciiDoc files. -Publishing to cassandra.apache.org :: Merging `asf-staging` to `asf-site` ------------------------------------ +### Generate the website and documentation using pre-generated Cassandra AsciiDoc in remote repositories -Updating the main website, after verifying the staged website, involves copying the `asf-staging` branch to `asf-site`. A normal git merge is not used, because the `asf-staging` is forced updated after each ci-cassandra.apache.org build. Instead make live the staged website by copying the `asf-staging` to the `asf-site` branch. +To build using a remote copy of the Cassandra repository that contains the generated AsciiDoc files, and the Cassandra Website run the following command. - git fetch origin - git switch asf-site - git reset --hard origin/asf-staging - git push -f origin asf-site +```bash +$ ./run.sh \ + website \ + build \ + -i \ + -u cassandra:https://github.com/my_orgranisation/cassandra.git \ + -u cassandra-website:https://github.com/my_orgranisation/cassandra-website.git +``` +You can have a combination of local and remote repository paths. For example, you could have a local copy of the Cassandra Website repository and want to use the remote Cassandra repository to include the current documentation when rendering the website. To do this you can run the following command. -## Building the Site UI +```bash +$ ./run.sh \ + website \ + build \ + -i \ + -u cassandra:https://github.com/my_orgranisation/cassandra.git \ + -u cassandra-website:/local/path/to/cassandra-website/repository +``` + +### Generate the website using local copy of the ui-bundle -### Executing Tasks +You can use your own *ui-bundle.zip* file containing the information on how to style the website when building it. The *ui-bundle.zip* file can be generated using the `./run.sh` script. See the [Building the Site UI](#building-the-site-ui) section for furher details on how to build the *ui-bundle.zip*. -To get a list of the tasks that can be executed run the following commands: +To supply your own *ui-bundle.zip* file when building the website, run the following command. +```bash +$ ./run.sh website build -z /local/path/to/ui-bundle.zip ``` -$ export BUILD_USER=build -$ docker-compose build website-ui: -$ docker-compose run website-ui + +The path to the *ui-bundle.zip* file can also be a remote URL. You can supply the URL using the following command. + +```bash +$ ./run.sh website build -z https://github.com/apache/cassandra-website/archive/refs/tags/ui-bundle-1.0.zip +``` + +The styling contained in the *ui-bundle.zip* will be applied to docs if they are being rendered as well. + +By default, the Docker container used to render the site will reference a GitHub release version of the *ui-bundle.zip*. + +## Building the Site UI + +To get a list of the tasks that can be executed, run the following commands. + +```bash +$ ./run.sh website-ui ``` A task can be executed using the following commands: ```bash -$ export BUILD_USER=build -$ docker-compose build website-ui: -$ docker-compose run website-ui <task> +$ ./run.sh website-ui <task> +``` + +A full list of tasks can be found by running the following command + +```bash +$ ./run.sh website-ui tasks ``` ### Building Bundle -TODO: Add info +Antora needs a *ui-bundle.zip* when rendering the website and documentation content. It contains CSS, Java Script, fonts, images and other assets which define the look and feel of the website. + +To generate the *ui-bundle.zip* from the assets in the *site-ui* directory, run the following command. + +```bash +$ ./run.sh website-ui bundle +``` + +This will build the UI Bundle using your local copy of the styling assets located in *./site-ui/*. -### Releasing Bundle +When packaged successfully, the *ui-bundle.zip* will be located in the *./site-ui/build/* directory of the repository. -TODO: Add info +:warning: *Tip:* In order to prevent root-owned modified files in your repository, the container executes operations as a non-root user. By default, the user is `build` and has the user and group permissions set to `UID=1000` and `GID=1000` respectfully. These permissions are usually the first user configured on a linux machine. + +If your local user has different user and group permissions you can set up the container user with your local UID:GID. In addition, you can set the build user in the container your local username. These changes can be made when building the container using the following command: + +```bash +$ ./run.sh website-ui container -a BUILD_USER_ARG:$(whoami) -a UID_ARG:$(id -u) -a GID_ARG:$(id -g) +``` + +If you need to customise the container user as noted above, you must do this before you build the UI Bundle or run any other website-ui command. ### Preview UI +An offline preview mode exists if you want to view example website content as you make changes to the UI. Preview mode can be launched using the following command. + ```bash -$ export BUILD_USER=build -$ docker-compose build website-ui-preview -$ docker-compose up website-ui-preview -``` \ No newline at end of file +$ ./run website-ui preview +``` + +The example content can be viewed on [http://localhost:5252](http://localhost:5252). + +While preview mode is running, any changes you make to the source files will be instantly reflected in the browser. This works by monitoring the project for changes, running the build task if a change is detected, and sending the updates to the browser. + +The files in the *preview-src/* folder provide the sample content that allow you to see the UI in action. In this folder, you will primarily find pages written in AsciiDoc. These pages provide a representative sample and kitchen sink of content from the real site. + +Press `Ctrl+C` to stop the preview server and end the continuous build. + +# Merging `asf-staging` to `asf-site` + +Updating the main website, after verifying the staged website, involves copying the `asf-staging` branch to `asf-site`. A normal git merge is not used, because the `asf-staging` is forced updated after each ci-cassandra.apache.org build. Instead make live the staged website by copying the `asf-staging` to the `asf-site` branch. + + git fetch origin + git switch asf-site + git reset --hard origin/asf-staging + git push -f origin asf-site + +# Details + +## Tooling components + +The tooling is made up of the following components + +* Run script: `./run.sh` - Provides a single commandline interface that generates the docker commands to run the website and UI docker containers. Using the containers, it can build the Cassandra website UI components, generate the Cassandra versioned documentation, and generate the website HTML. +* Website Docker container: `site-content/Dockerfile` and `site-content/docker-entrypoint.sh` - Contains the libraries necessary to generate the Cassandra versioned documentation, and generate the website HTML using Antora. +* Antora Site YAML script: `bin/site_yaml_generator.py` - Used by the Website Docker container to create the YAML file that defines the sources for the website content, optionally the cassandra versioned documentation, and the UI styling bundle to apply. +* Website UI Docker container: `site-ui/Dockerfile` and `site-ui/docker-entrypoint.sh` - Contains the libraries necessary to generate the UI bundle ZIP file the is applied by Antora to style the website and documentation. + +## Why is Antora being used + +Antora is being used for the website generation because it is designed to create websites that have version documentation. For example, when a new version of Cassandra is released, a new version of the documentation will be generated as well. Hence, if there is a change in the behaviour of Cassandra or a tool in the project, it will be captured in the latest version of the documentation. Users of the project will have the ability to select the version of the documentation they are intere [...] + +## Why is the styling separated from the content + +Separating the layout/style, and the content means that stying changes can be made with little to no impact on content and vice-versa. In addition, changes to the styling can happen in parallel while website content is updated without conflict. \ No newline at end of file --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
