This is an automated email from the ASF dual-hosted git repository.
michaelsembwever pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/cassandra-website.git
The following commit(s) were added to refs/heads/trunk by this push:
new 30bdcc103 Update testing and CI documentation
30bdcc103 is described below
commit 30bdcc103ba3932d7dc2f43c7e016c339d81d633
Author: mck <[email protected]>
AuthorDate: Wed Aug 19 16:05:06 2026 +0200
Update testing and CI documentation
Nothing has been updated after this ml post, with significantly changed the
status quo
https://lists.apache.org/thread/8pjk481f59rtj18v8d95bojxwmzp9jdr
patch by Mick Semb Wever; reviewed by Jon Haddad for CASSANDRA-21586
---
.../source/modules/ROOT/pages/development/ci.adoc | 248 ++++++++++++---------
.../ROOT/pages/development/how_to_review.adoc | 5 +-
.../modules/ROOT/pages/development/index.adoc | 2 +-
.../modules/ROOT/pages/development/patches.adoc | 23 +-
.../modules/ROOT/pages/development/testing.adoc | 72 ++----
5 files changed, 181 insertions(+), 169 deletions(-)
diff --git a/site-content/source/modules/ROOT/pages/development/ci.adoc
b/site-content/source/modules/ROOT/pages/development/ci.adoc
index 17b54e5a1..f43fcaa4d 100644
--- a/site-content/source/modules/ROOT/pages/development/ci.adoc
+++ b/site-content/source/modules/ROOT/pages/development/ci.adoc
@@ -1,104 +1,148 @@
:page-layout: basic
-== CI Environments
-
-=== About CI testing and Apache Cassandra
-
-Cassandra can be automatically tested using various test suites, that
-are either implemented based on JUnit or the
-https://github.com/apache/cassandra-dtest[dtest] scripts written in
-Python. As outlined in `testing`, each kind of test suite addresses a
-different way to test Cassandra. Eventually, all of the tests will be
-executed together on the CI platform at
-https://builds.apache.org[builds.apache.org], running
-http://jenkins-ci.org[Jenkins].
-
-=== Setting up your own Jenkins server
-
-Jenkins is an open source solution that can be installed on a large
-number of platforms. Setting up a custom Jenkins instance for Cassandra
-may be desirable for users who have hardware to spare, or organizations
-that want to run Cassandra tests for custom patches before contribution.
-
-Please refer to the Jenkins download and documentation pages for details
-on how to get Jenkins running, possibly also including slave build
-executor instances. The rest of the document will focus on how to setup
-Cassandra jobs in your Jenkins environment.
-
-==== Required plugins
-
-In addition, the following plugins need to be installed along with the standard
-plugins (git, ant, ..).
-
-You can install any missing plugins using the install manager.
-
-Go to `Manage Jenkins -> Manage Plugins -> Available` and install the
-following plugins and respective dependencies:
-
-* Job DSL
-* Javadoc Plugin
-* description setter plugin
-* Throttle Concurrent Builds Plug-in
-* Test stability history
-* Post Build Script
-
-==== Setup seed job
-
-. Config `New Item`
-
-* Name it `Cassandra-Job-DSL`
-* Select `Freestyle project`
-
-. Under `Source Code Management` select Git using the repository:
-`https://github.com/apache/cassandra-builds`
-
-. Under `Build`, confirm `Add build step` -> `Process Job DSLs` and enter
-at `Look on Filesystem`: `jenkins-dsl/cassandra_job_dsl_seed.groovy`
-
-Generated jobs will be created based on the Groovy script's default
-settings. You may want to override settings by checking
-`This project is parameterized` and add `String Parameter` for on the
-variables that can be found in the top of the script. This will allow
-you to setup jobs for your own repository and branches (e.g. working
-branches).
-
-[arabic, start=4]
-. When done, confirm "Save".
-
-You should now find a new entry with the given name in your project
-list. However, building the project will still fail and abort with an
-error message "Processing DSL script
-cassandra_job_dsl_seed.groovy ERROR: script not yet approved for use".
-Go to `Manage Jenkins` -> `In-process Script Approval` to fix this issue.
-Afterwards you should be able to run the script and have it generate
-numerous new jobs based on the found branches and configured templates.
-
-Jobs are triggered by either changes in Git or are scheduled to execute
-periodically, e.g. on daily basis.
-Jenkins will use any available executor with the label "cassandra", once the
job
-is to be run.
-Please make sure to make any executors available by selecting
-`Build Executor Status` -> `Configure` -> Add "`cassandra`" as label and
-save.
-
-Executors need to have "JDK 1.8 (latest)" installed. This is done under
-`Manage Jenkins -> Global Tool Configuration -> JDK Installations…`.
-Executors also need to have the `virtualenv` package installed on their
-system.
-
-=== CircleCI
-
-Cassandra ships with a default https://circleci.com[CircleCI]
-configuration to enable running tests on your branches.
-Go to the CircleCI website, click "Login" and log in with your github account.
-Then give CircleCI permission to watch your repositories.
-
-Once you have done that, you can optionally configure CircleCI to run
-tests in parallel if you wish:
-
-[arabic,start=1]
-. Click `Projects` and select your github account, and then click the settings
for your project.
-. Set the parallelism setting. If you leave the default value of 1
-for Cassandra, only `ant eclipse-warnings` and `ant test` will be run.
-If you change the value to 4, Circle CI also runs `ant long-test`,
-`ant test-compression` and `ant stress-test`.
+== Continuous Integration
+
+The project has three separate CI surfaces. Knowing which one applies saves a
lot of time.
+
+[cols="1,2,2",options="header",]
+|===
+|System |What it runs |When it runs
+
+|GitHub Actions
+|`ant check`: licence (rat) and checkstyle checks. +
+Changes under `.build/run-ci*` or `.jenkins/` have their own checks.
+|Automatically, on every push to any branch of any fork
+
+|Pre-commit CI: +
+https://pre-ci.cassandra.apache.org[pre-ci.cassandra.apache.org], +
+or any clone of it
+|The test pipeline against your branch, at the profile you choose
+|On demand, via UI, or with `.build/run-ci`
+
+|Post-commit CI: +
+https://ci-cassandra.apache.org[ci-cassandra.apache.org]
+|The full `post-commit` pipeline against the official branches
+|Automatically, after every push on an `apache/cassandra` branch
+|===
+
+Testing a patch before it is committed is the author's and the reviewer's job,
and it is done with pre-commit CI.
+
+[#pre_commit_ci]
+=== Pre-commit CI
+
+Pre-commit CI is a clone of ci-cassandra.apache.org that runs in any
Kubernetes cluster. The Jenkinsfile, the docker images, and the test scripts
are the same ones used post-commit and locally, so a failure seen in one place
is reproducible in the others.
+
+You have three ways to get at one:
+
+* https://pre-ci.cassandra.apache.org[pre-ci.cassandra.apache.org], hosted by
the ASF and paid for by a donation of AWS credits from Amazon. It is intended
for committers and trusted contributors who do not have access to a
company-private CI clone. Ask on the
https://lists.apache.org/list.html?\[email protected][dev mailing
list], or contact a PMC member, for an account.
+* Your own cluster, provisioned with `.build/run-ci --only-setup`. See
<<provisioning_your_own>>.
+* A clone operated by your employer. Several companies run one; the command
line and the results are identical.
+
+Contributors without an account should say so on the ticket. The reviewer or
committer then runs CI on the patch's behalf.
+
+==== Using `run/ci` – One-time setup
+
+The `.build/run-ci` script needs its Python dependencies:
+
+[source,plaintext]
+----
+pip install -r .build/run-ci.d/requirements.txt
+----
+
+`helm` and `kubectl` are needed only when provisioning a cluster, not when
submitting builds to an existing Jenkins.
+
+The Jenkins URL and user can be kept in `.build/.run-ci.env` (or in the
`JENKINS_URL` and `JENKINS_USER` environment variables) instead of being passed
on every invocation. The script prompts for the password.
+
+==== Using `run/ci` – Running a build
+
+The script takes the branch of a *pushed* fork, not your working tree; commit
and push first. It defaults to the current branch and its tracking remote.
+
+[source,plaintext]
+----
+# run the current branch, on the cluster in your current kubeconfig context
+.build/run-ci
+
+# run the current branch on a Jenkins you have no kubectl access to
+.build/run-ci --url pre-ci.cassandra.apache.org --user myuser
+
+# a full pre-commit run of someone else's branch
+.build/run-ci -r https://github.com/jrwest/cassandra.git -b jwest/15452-5.0 -p
pre-commit
+
+# only the jvm-dtest and stress stages, on JDK 11
+.build/run-ci -p custom -e 'stress.*|jvm-dtest.*' -j 11
+----
+
+`.build/run-ci --help`, and
https://github.com/apache/cassandra/blob/trunk/.build/run-ci.d/README.md[.build/run-ci.d/README.md],
list every option.
+
+==== Profiles
+
+A profile is a named set of pipeline stages. The definitive list is
`pipelineProfiles()` in
+https://github.com/apache/cassandra/blob/trunk/.jenkins/Jenkinsfile[.jenkins/Jenkinsfile].
+
+[cols="1,3",options="header",]
+|===
+|Profile |Purpose
+
+|`skinny`
+|The default. Lint, unit tests, cqlsh tests, jvm-dtests and dtests: enough to
catch the obvious, and quick.
+
+|`packaging`
+|Artifacts, deb and rpm packages, lint. Used when only the build or packaging
changed.
+
+|`pre-commit`
+|What a patch is expected to pass before it is committed.
+
+|`pre-commit w/ upgrades`
+|As above, plus the upgrade tests. Required whenever the patch could affect
upgrades, serialization, or on-disk formats.
+
+|`post-commit`
+|Everything: 140–200k tests, including the large and novnode dtests. This is
what runs on ci-cassandra.apache.org.
+
+|`custom`
+|The stages matching the regexp given with `-e`. Useful for iterating on one
failure.
+|===
+
+Wall-clock time is a function of how far the cluster can scale out, not of the
profile alone; a `post-commit` run finishes within a few hours on a cluster
that can reach a few hundred nodes.
+
+==== Results
+
+On completion the script writes two files under `build/ci/`:
+
+* `ci_summary_<fork>_<branch>_<build>.html`, an aggregated summary of the run
+* `results_details_<fork>_<branch>_<build>.tar.xz`, the individual test results
+
+Attach both to the Jira ticket. They are the record that the patch was
tested, they outlive any CI instance, and reviewers can read them instead of
having to browse your Jenkins instance (which might even be private). Use
`.build/run-ci -o <build number>` to fetch the artefacts of an earlier build.
+
+[#provisioning_your_own]
+==== Provisioning your own instance
+
+`.build/run-ci --only-setup` installs the Jenkins helm chart into the current
Kubernetes context. With this approach you never have to open the Jenkins UI,
and the cluster scales back to a single controller node when idle, so the cost
tracks what you use.
+
+The one prerequisite is a cluster whose node groups match the agent sizes the
pipeline asks for; the commands for GKE and EKS are in
+https://github.com/apache/cassandra/blob/trunk/.jenkins/k8s/README.md[.jenkins/k8s/README.md].
+
+Tear it down with `.build/run-ci --only-tear-down`.
+
+==== Reproducing a failure locally
+
+Because CI runs the in-tree scripts and images, any stage can be run on a
laptop:
+
+[source,plaintext]
+----
+.build/docker/run-tests.sh -a test -c 1/64 -j 17
+.build/docker/run-tests.sh -a jvm-dtest -t BooleanTest
+----
+
+See
https://github.com/apache/cassandra/blob/trunk/.build/README.md[.build/README.md]
and xref:development/testing.adoc[testing].
+
+[#post_commit_ci]
+=== Post-commit CI
+
+https://ci-cassandra.apache.org[ci-cassandra.apache.org] runs the
`post-commit` profile against trunk and each supported release branch, on
hardware donated by NetApp, DataStax, Huawei and iland. Its results are
archived at
https://nightlies.apache.org/cassandra/[nightlies.apache.org/cassandra], and
https://butler.cassandra.apache.org[Butler] shows them over time, which is how
you tell a new regression from a long-standing flaky test.
+
+When a post-commit failure is not obviously your patch, check Butler and the
issue tracker before assuming it is; when it is, a Jira ticket for the failure
is expected.
+
+=== Further reading
+
+* https://lists.apache.org/thread/8pjk481f59rtj18v8d95bojxwmzp9jdr[ANNOUNCE:
Pre-commit CI @ pre-ci.cassandra.apache.org], the announcement of the AWS
donation and of this process
+* https://issues.apache.org/jira/browse/CASSANDRA-18137[CASSANDRA-18137], the
epic that made ci-cassandra.apache.org repeatable, and
https://issues.apache.org/jira/browse/CASSANDRA-18145[CASSANDRA-18145],
provisioning it in any Kubernetes cluster
diff --git
a/site-content/source/modules/ROOT/pages/development/how_to_review.adoc
b/site-content/source/modules/ROOT/pages/development/how_to_review.adoc
index d31ab8e4f..887ba3ae0 100644
--- a/site-content/source/modules/ROOT/pages/development/how_to_review.adoc
+++ b/site-content/source/modules/ROOT/pages/development/how_to_review.adoc
@@ -58,8 +58,9 @@ CqlTester methods) or abstract it there for reuse?
* If the code may be affected by multi-node clusters, are there dtests?
* If the code may take a long time to test properly, are there CVH
tests?
-* Is the test passing on CI for all affected branches (up to trunk, if
-applicable)? Are there any regressions?
+* Are the xref:development/ci.adoc[pre-commit CI] results attached to
+the ticket, for all affected branches (up to trunk, if applicable)? Are
+there any regressions?
* If patch affects read/write path, did we test for performance
regressions w/multiple workloads?
* If adding a new feature, were tests added and performed confirming it
diff --git a/site-content/source/modules/ROOT/pages/development/index.adoc
b/site-content/source/modules/ROOT/pages/development/index.adoc
index 7d1664e13..1e9f9e768 100644
--- a/site-content/source/modules/ROOT/pages/development/index.adoc
+++ b/site-content/source/modules/ROOT/pages/development/index.adoc
@@ -8,7 +8,7 @@
* xref:development/how_to_commit.adoc[How to commit]
* xref:development/how_to_review.adoc[How to review]
* xref:development/patches.adoc[Patches]
-* xref:development/ci.adoc[CI]
+* xref:development/ci.adoc[Continuous Integration]
* xref:development/dependencies.adoc[Dependencies]
* xref:development/documentation.adoc[Documentation]
* xref:development/release_process.adoc[Release process]
diff --git a/site-content/source/modules/ROOT/pages/development/patches.adoc
b/site-content/source/modules/ROOT/pages/development/patches.adoc
index 13956ac20..be9779606 100644
--- a/site-content/source/modules/ROOT/pages/development/patches.adoc
+++ b/site-content/source/modules/ROOT/pages/development/patches.adoc
@@ -146,12 +146,17 @@ using https://git-scm.com/docs/git-worktree[Git
worktrees] feature.
. Make sure all tests (including yours) pass using ant as described in
xref:development/testing.adoc[testing]. If you suspect a test failure
is unrelated to your change, it may be useful to check the test's
-status by searching the issue tracker or looking at
-https://builds.apache.org/[CI] results for the relevant upstream
-version. Note that the full test suites take many hours to complete,
-so it is common to only run specific relevant tests locally before
-uploading a patch. Once a patch has been uploaded, the reviewer or
-committer can help setup CI jobs to run the full test suites.
+status by searching the issue tracker or looking at the
+xref:development/ci.adoc[post-commit CI] results for the relevant
+upstream version. Note that the full test suites take many hours to
+complete, so it is common to only run specific relevant tests locally
+before uploading a patch.
+. Run the patch through xref:development/ci.adoc[pre-commit CI], with
+`.build/run-ci`, and attach the `ci_summary` and `results_details`
+artefacts to the ticket. This is expected of every patch before it is
+committed. Contributors without access to a CI system should say so on
+the ticket; the reviewer or committer then runs CI on the patch's
+behalf.
. Consider going through the xref:development/how_to_review.adoc[how
to review] page for your code. This will help you to understand how
others will consider your change for inclusion.
@@ -164,7 +169,7 @@ stage, especially for incremental review, but once +1d, do
either:
branch), or
. Squash the commits in-place in your branches into one
-[arabic, start=6]
+[arabic, start=7]
. Include a CHANGES.txt entry (put it at the top of the list), and
format the commit message appropriately in your patch as below. Please
note that only user-impacting items
@@ -179,7 +184,7 @@ is necessary.
<Optional lengthier description>
patch by <Authors>; reviewed by <Reviewers> for CASSANDRA-#####
----
-[arabic, start=7]
+[arabic, start=8]
. When you're happy with the result, create a patch. We suggest that
you use a similar format (note blank lines) for the commit log message:
@@ -213,7 +218,7 @@ push your branch:
git push --set-upstream origin 12345-3.0
----
-[arabic, start=8]
+[arabic, start=9]
. To make life easier for your reviewer/committer, you may want to make
sure your patch applies cleanly to later branches and create additional
patches/branches for later Cassandra versions to which your original
diff --git a/site-content/source/modules/ROOT/pages/development/testing.adoc
b/site-content/source/modules/ROOT/pages/development/testing.adoc
index cf5f74576..343b4e16d 100644
--- a/site-content/source/modules/ROOT/pages/development/testing.adoc
+++ b/site-content/source/modules/ROOT/pages/development/testing.adoc
@@ -39,9 +39,9 @@ Cassandra version agnostic. They include upgrade tests.
In case you want to run DTests with your own version of CCM, please refer to
requirements.txt in
https://github.com/apache/cassandra-dtest[apache/cassandra-dtest] how to do it.
-The recipes for running those tests can be found in the cassandra-builds
repository
https://github.com/apache/cassandra-builds/tree/trunk/build-scripts[here].
+The recipes for running those tests are the scripts under `.build/`, in the
Cassandra repository itself; these are the same scripts CI runs. See
https://github.com/apache/cassandra/blob/trunk/.build/README.md[.build/README.md].
-Running full test suites locally takes hours, if not days. Beyond running
specific tests you know are applicable, or are failing, to the work at hand, it
is recommended to rely upon the project's
https://cassandra.apache.org/_/development/ci.html[Continuous Integration
systems]. If you are not a committer, and don't have access to a premium
CircleCI plan, ask one of the committers to test your patch on the project's
https://ci-cassandra.apache.org/[ci-cassandra.apache.org].
+Running full test suites locally takes hours, if not days. Beyond the
specific tests that bear on the work at hand, rely upon the project's
xref:development/ci.adoc[CI]. Push your branch to your fork, run
`.build/run-ci`, and attach the resulting summary and details artefacts to the
ticket. Contributors with neither their own CI system nor an account on
https://pre-ci.cassandra.apache.org[pre-ci.cassandra.apache.org] should say so
on the ticket, and a committer will run CI on the patc [...]
[#java_tests]
== Java tests
@@ -411,68 +411,30 @@ parameters). The list of all options can be shown by
running
The Docker approach is recommended for running Python distributed tests.
The behavior will be more repeatable, matching the same environment as
-the official testing on https://ci-cassandra.apache.org/[Cassandra CI].
+the project's xref:development/ci.adoc[CI], which runs the very same
+image and scripts.
-==== Setup Docker
-
-If you are on Linux, you need to install Docker using the system package
-manager.
-
-If you are on MacOS, you can use either
-https://www.docker.com/products/docker-desktop[Docker Desktop] or some
-https://runnable.com/docker/install-docker-on-macos[other approach].
-
-==== Pull the Docker image
-
-The Docker image used on the official Cassandra CI can be found in
-https://github.com/apache/cassandra-builds[this] repository.
-You can use either
-https://github.com/apache/cassandra-builds/blob/trunk/docker/testing/ubuntu2004_j11.docker[docker/testing/ubuntu2004_j11.docker]
-or
-https://github.com/apache/cassandra-builds/blob/trunk/docker/testing/ubuntu2004_j11_w_dependencies.docker[docker/testing/ubuntu2004_j11_w_dependencies.docker]
-The second choice has prefetched dependencies for building each main
-Cassandra branch. Those images can be either built
-locally (as per instructions in the GitHub repo) or pulled from the
-Docker Hub - see
-https://hub.docker.com/search?q=apache%2Fcassandra-testing&type=image[here].
-
-First, pull the image from Docker Hub (it will either fetch or
-update the image you previously fetched):
+The simplest route is to let the wrapper script do the work; it builds
+or pulls the image, mounts your working tree, and runs the given test
+target:
[source,plaintext]
----
-docker pull apache/cassandra-testing-ubuntu2004-java11-w-dependencies
+.build/docker/run-tests.sh -a dtest
+.build/docker/run-tests.sh -a dtest -t "test_bootstrap"
----
-==== Start the container
+The rest of this section covers working inside the container by hand,
+which is useful when iterating on a single test.
-[source,plaintext]
-----
-docker run -di -m 8G --cpus 4 \
---mount
type=bind,source=/path/to/cassandra/project,target=/home/cassandra/cassandra \
---mount
type=bind,source=/path/to/cassandra-dtest,target=/home/cassandra/cassandra-dtest
\
---name test \
-apache/cassandra-testing-ubuntu2004-java11-w-dependencies \
-dumb-init bash
-----
-
-[TIP]
-.Hint
-====
-Many distributed tests are not that demanding in terms of resources
-- 4G / 2 cores should be enough to start one node. However, some tests
-really run multiple nodes, and some of them are automatically skipped
-if the machine has less than 32G (there is a way to force running them
-though). Usually 8G / 4 cores is a convenient choice which is enough
-for most of the tests.
-====
+==== Setup Docker
-To log into the container, use the following `docker exec` command:
+If you are on Linux, you need to install Docker using the system package
+manager.
-[source,plaintext]
-----
-docker exec -it `docker container ls -f name=test -q` bash
-----
+If you are on MacOS, you can use either
+https://www.docker.com/products/docker-desktop[Docker Desktop] or some
+https://runnable.com/docker/install-docker-on-macos[other approach].
[#setup_python_env]
==== Setup Python environment
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
