This is an automated email from the ASF dual-hosted git repository.
adoroszlai pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/ozone.git
The following commit(s) were added to refs/heads/master by this push:
new 1da0bd8 HDDS-5609. Document CI workflow (#2607)
1da0bd8 is described below
commit 1da0bd8714638fdbd5fca798f65827ca24eff0f6
Author: GeorgeJahad <[email protected]>
AuthorDate: Fri Oct 1 09:36:35 2021 -0700
HDDS-5609. Document CI workflow (#2607)
---
.github/ci.md | 102 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++
CONTRIBUTING.md | 2 ++
2 files changed, 104 insertions(+)
diff --git a/.github/ci.md b/.github/ci.md
new file mode 100644
index 0000000..4fa3fef
--- /dev/null
+++ b/.github/ci.md
@@ -0,0 +1,102 @@
+# Ozone CI with Github Actions
+
+The Ozone project uses Github Actions, (GA), for its CI system. GA are
implemented with workflows, which are groups of *jobs* combined to accomplish a
CI task, all defined in a single yaml file. The Ozone workflow yaml files are
[here](./workflows).
+
+## Workflows
+
+### build-branch Workflow
+This is the most important [workflow](./workflows/post-commit.yml). It runs
the tests that verify the latest commits.
+
+It is triggered each time a pull request is created or synchronized, (ie when
the remote branch is pushed to.) It is also "scheduled" on the master branch
twice a day, (00:30 and 12:30). (Those are the runs
[here](https://github.com/apache/ozone/actions/workflows/post-commit.yml?query=event%3Aschedule++)
which are marked "scheduled", and have no branch label.)
+
+The build-branch workflow is divided into a number of different jobs, most of
which run in parallel. Each job is described below.
+
+Some of the jobs are defined using GA's "build matrix" feature. This allows
you define similar jobs with a single job definition. Any differences are
specified by a list of values for a specific key. For example, the "compile"
job uses the matrix feature to generate the images with different versions of
java. There, the matrix is specified by the "java" key which has a list of
values describing which version of java to use, (8 or 11.)
+
+The jobs currently using the "build matrix" feature are: "compile", "basic",
"acceptance" and "integration". These jobs also use GA's fail-fast flag to
cancel the other jobs in the same matrix, if one fails. For example, in the
"compile" job, if the java 8 build fails, the java 11 build will be cancelled
due to this flag, but the other jobs outside the "compile" matrix are
unaffected.
+
+While the fail-fast flag only works within a matrix job, the "Cancelling"
workflow, (described below,) works across jobs.
+
+
+#### build-info job
+
+[The build-info job script](../dev-support/ci/selective_ci_checks.sh) runs
before the others and determines which of the other jobs are to be run. If the
workflow was triggered by some event other than a PR, then all jobs/tests are
run. They are also all run if the PR has a label containing the following
string, "full tests needed".
+
+Otherwise, *build-info* first generates a list of files that were changed by
the PR. It matches that list against a series of regex's, each of which is
associated with a different job. It sets the appropriate flag for each match.
Those boolean flags are used later in the run to decide whether the
corresponding job should be run
+
+For example, a regex like the following is used to determine if the Kubernetes
flag should be set.
+```
+ local pattern_array=(
+ "^hadoop-ozone/dev-support/checks/kubernetes.sh"
+ "^hadoop-ozone/dist/src/main/k8s"
+ )
+```
+
+
+
+#### compile job
+[Builds](../hadoop-ozone/dev-support/checks/build.sh) the Java 8 and 11
versions of the jars, and saves the java 8 version for some of the subsequent
jobs.
+
+#### basic job
+Runs a subset of the following subjobs depending on what was selected by
build-info
+- author: [Verifies](../hadoop-ozone/dev-support/checks/author.sh) none of the
Java files contain the @author annotation
+- bats: [Checks](../hadoop-ozone/dev-support/checks/bats.sh) bash scripts,
(using the [Bash Automated Testing
System](https://github.com/bats-core/bats-core#bats-core-bash-automated-testing-system-2018))
+- checkstyle: [Runs](../hadoop-ozone/dev-support/checks/checkstyle.sh) 'mvn
checkstyle' plugin to confirm Java source abides by Ozone coding conventions
+- docs: [Builds](../hadoop-ozone/dev-support/checks/docs.sh) website with
[Hugo](https://gohugo.io/)
+- findbugs: [Runs](../hadoop-ozone/dev-support/checks/findbugs.sh) spotbugs
static analysis on bytecode
+- rat (release audit tool):
[Confirms](../hadoop-ozone/dev-support/checks/rat.sh) source files include
licenses
+- unit: [Runs](../hadoop-ozone/dev-support/checks/unit.sh) 'mvn test' for all
non integration tests
+
+#### dependency job
+[Confirms](../hadoop-ozone/dev-support/checks/dependency.sh) that the list of
jars included in the current build matches the expected ones defined
[here](../hadoop-ozone/dist/src/main/license/jar-report.txt)
+
+If they don't match, it describes how to make the updates to include the
changes, (if they are intentional). Otherwise, the changes should be removed.
+
+#### acceptance job
+[Runs](../hadoop-ozone/dev-support/checks/acceptance.sh) smoketests using
robot framework and a real docker compose cluster. There are three iterations,
"secure", "unsecure", and "misc", each running in parallel, as different matrix
configs.
+
+#### kubernetes job
+[Runs](../hadoop-ozone/dev-support/checks/kubernetes.sh) k8s tests
+
+#### integration job
+[Runs](../hadoop-ozone/dev-support/checks/integration.sh) 'mvn test' for all
integration/minicluster tests, split into multiple subjobs, by a matrix config.
+
+#### coverage job
+[Merges](../hadoop-ozone/dev-support/checks/coverage.sh) the coverage data
from the following jobs that were run earlier:
+- acceptance
+- basic
+- integration
+
+### Cancelling Workflow
+[This](./workflows/cancel-ci.yaml) workflow is triggered each time a
[build-branch](ci.md#build-branch-workflow) workflow is triggered. It reduces
GA usage, by cancelling PR workflows that have a job failure. Specifically, it
checks all PR workflows running at the time it is invoked. It cancels any of
those which have a failed job, (at that time). Any PR workflows with jobs that
fail after that will be caught by a subsequent run of the "Cancelling" workflow.
+
+Note that it checks the status of all PR workflows running at that time, not
just the one that triggered it.
+
+### close-prs Workflow
+[This](./workflows/close-pending.yaml) workflow is scheduled each night at
midnight; it closes PR's that have not been updated in the last 21 days, while
letting the author know they are free to reopen.
+
+### comment-commands Workflow
+[This](./workflows/comments.yaml) workflow is triggered each time a comment is
added/edited to a PR. It checks to see if the body of the comment begins with
one of the following strings and, if so, invokes the corresponding command.
+- /close : [Close](./comment-commands/close.sh) pending pull request (with
message saying author is free to reopen.)
+- /help : [Show](./comment-commands/help.sh) all the available comment commands
+- /label : [Add](./comment-commands/label.sh) new label to the issue: /label
"label"
+- /pending : [Add](./comment-commands/pending.sh) a REQUESTED_CHANGE type
review to mark issue non-mergeable: /pending "reason"
+- /ready : [Dismiss](./comment-commands/ready.sh) all the blocking reviews
+- /retest : [Provide](./comment-commands/retest.sh) help on how to trigger new
CI build
+
+
+## Old/Deprecated Workflows
+The following workflows no longer run but still exist on the
[actions](https://github.com/apache/ozone/actions) page for historical reasons:
+- [Build](https://github.com/apache/ozone/actions/workflows/main.yml)
+- [build-branch](https://github.com/apache/ozone/actions/workflows/chaos.yml)
+- [pr-check](https://github.com/apache/ozone/actions/workflows/pr.yml)
+
+Note that the deprecated
[build-branch](https://github.com/apache/ozone/actions/workflows/chaos.yml) has
the same name as the current
[build-branch](https://github.com/apache/ozone/actions/workflows/post-commit.yml).
(They can be distinguished by the URL.)
+
+
+## Tips
+
+- When a build of the Ozone master branch fails, its artifacts are stored
[here](https://elek.github.io/ozone-build-results/).
+- To trigger rerunning the tests, push a commit like this to your PR: ```git
commit --allow-empty -m 'trigger new CI check'```
+- [This
wiki](https://cwiki.apache.org/confluence/display/OZONE/Running+Ozone+Smoke+Tests+and+Unit+Tests)
contains tips on running tests locally.
+- [This
wiki](https://cwiki.apache.org/confluence/display/OZONE/Github+Actions+tips+and+tricks)
contains tips on special handling of the CI system, such as "Executing one
test multiple times", or "ssh'ing in to the CI machine while the tests are
running".
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
index eb5c2b2..0dba193 100644
--- a/CONTRIBUTING.md
+++ b/CONTRIBUTING.md
@@ -192,3 +192,5 @@ IntelliJ may not pick up protoc generated classes as they
can be very huge. If t
3. Restart your IDE
+## CI
+The Ozone project uses Github Actions for its CI system. The configuration is
described in detail [here](.github/ci.md).
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
