GeorgeJahad commented on a change in pull request #2607: URL: https://github.com/apache/ozone/pull/2607#discussion_r700728974
########## File path: .github/ci.md ########## @@ -0,0 +1,94 @@ +# Ozone CI with Github Actions + +The Ozone project uses Github Actions, (GA), for its CI system. GA are implemented with workflows, which are groups of *jobs* combined to accomplish a CI task, all defined in a single yaml file. The Ozone workflow yaml files are [here](./workflows). + +## Workflows + +### build-branch Workflow +This is the most important [workflow](./workflows/post-commit.yml). It runs the tests that verify the latest commits. + +It is triggered each time a pull request is created or synchronized, (ie when the remote branch is pushed to.) It is also "scheduled" on the master branch twice a day, (00:30 and 12:30). (Those are the runs [here](https://github.com/apache/ozone/actions/workflows/post-commit.yml?query=event%3Aschedule++) which are marked "scheduled", and have no branch label.) + +The build-branch workflow is divided into a number of different jobs, most of which run in parallel. Each of the jobs, (except "build-info" and "compile",) runs a subset of the test suite. They are all described below. + +#### build-info job + +[The build-info job script](../dev-support/ci/selective_ci_checks.sh) runs before the others and determines which of the other jobs are to be run. If the workflow was triggered by some event other than a PR, then all jobs/tests are run. They are also all run if the PR has a label containing the following string, "full tests needed". + +Otherwise, *build-info* first generates a list of files that were changed by the PR. It matches that list against a series of regex's, each of which is associated with a different job. It sets the appropriate flag for each match. Those boolean flags are used later in the run to decide whether the corresponding job should be run + +For example, a regex like the following is used to determine if the Kubernetes flag should be set. +``` + local pattern_array=( + "^hadoop-ozone/dev-support/checks/kubernetes.sh" + "^hadoop-ozone/dist/src/main/k8s" + ) +``` + + + +#### compile job +[Builds](../hadoop-ozone/dev-support/checks/build.sh) the Java 8 and 11 versions of the jars, and saves the java 8 version for some of the subsequent jobs. + +#### basic job +Runs a subset of the following subjobs depending on what was selected by build-info +- author: [Verifies](../hadoop-ozone/dev-support/checks/author.sh) none of the Java files contain the @author annotation +- bats: [Checks](../hadoop-ozone/dev-support/checks/bats.sh) bash scripts, (using the [Bash Automated Testing System](https://github.com/bats-core/bats-core#bats-core-bash-automated-testing-system-2018)) +- checkstyle: [Runs](../hadoop-ozone/dev-support/checks/checkstyle.sh) 'mvn checkstyle' plugin to confirm Java source abides by Ozone coding conventions +- docs: [Builds](../hadoop-ozone/dev-support/checks/docs.sh) website with [Hugo](https://gohugo.io/) +- findbugs: [Runs](../hadoop-ozone/dev-support/checks/findbugs.sh) spotbugs static analysis on bytecode +- rat (release audit tool): [Confirms](../hadoop-ozone/dev-support/checks/rat.sh) source files include licenses +- unit: [Runs](../hadoop-ozone/dev-support/checks/unit.sh) 'mvn test' for all non integration tests + +#### dependency job +[Confirms](../hadoop-ozone/dev-support/checks/dependency.sh) hadoop-ozone/dist/src/main/license/bin/LICENSE.txt is up to date, (includes references to all the latest jar files). + +#### acceptance job +[Runs](../hadoop-ozone/dev-support/checks/acceptance.sh) smoketests using robot framework and a real docker compose cluster + +#### kubernetes job +[Runs](../hadoop-ozone/dev-support/checks/kubernetes.sh) k8s tests + +#### integration job +[Runs](../hadoop-ozone/dev-support/checks/integration.sh) 'mvn test' for all integration/minicluster tests, split into 3 subjobs: +- client +- filesystem-hdds +- ozone + +#### coverage job +[Merges](../hadoop-ozone/dev-support/checks/coverage.sh) the coverage data from the following jobs that were run earlier: +- acceptance +- basic +- integration + +### Cancelling Workflow +[This](./workflows/cancel-ci.yaml) workflow is triggered each time a [build-branch](ci.md#build-branch-workflow) workflow is triggered. It monitors all the jobs triggered by that workflow run for failure and cancels any continuing jobs after one fails. This reduces our GA usage. + Review comment: @adoroszlai This cancelling functionality seems to overlap with the fail-fast flag that we are already using in post-commit.yml. Can you clarify the difference between the two? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
