GeorgeJahad commented on a change in pull request #2607: URL: https://github.com/apache/ozone/pull/2607#discussion_r708507187
########## File path: .github/ci.md ########## @@ -0,0 +1,94 @@ +# Ozone CI with Github Actions + +The Ozone project uses Github Actions, (GA), for its CI system. GA are implemented with workflows, which are groups of *jobs* combined to accomplish a CI task, all defined in a single yaml file. The Ozone workflow yaml files are [here](./workflows). + +## Workflows + +### build-branch Workflow +This is the most important [workflow](./workflows/post-commit.yml). It runs the tests that verify the latest commits. + +It is triggered each time a pull request is created or synchronized, (ie when the remote branch is pushed to.) It is also "scheduled" on the master branch twice a day, (00:30 and 12:30). (Those are the runs [here](https://github.com/apache/ozone/actions/workflows/post-commit.yml?query=event%3Aschedule++) which are marked "scheduled", and have no branch label.) + +The build-branch workflow is divided into a number of different jobs, most of which run in parallel. Each of the jobs, (except "build-info" and "compile",) runs a subset of the test suite. They are all described below. + +#### build-info job + +[The build-info job script](../dev-support/ci/selective_ci_checks.sh) runs before the others and determines which of the other jobs are to be run. If the workflow was triggered by some event other than a PR, then all jobs/tests are run. They are also all run if the PR has a label containing the following string, "full tests needed". + +Otherwise, *build-info* first generates a list of files that were changed by the PR. It matches that list against a series of regex's, each of which is associated with a different job. It sets the appropriate flag for each match. Those boolean flags are used later in the run to decide whether the corresponding job should be run + +For example, a regex like the following is used to determine if the Kubernetes flag should be set. +``` + local pattern_array=( + "^hadoop-ozone/dev-support/checks/kubernetes.sh" + "^hadoop-ozone/dist/src/main/k8s" + ) +``` + + + +#### compile job +[Builds](../hadoop-ozone/dev-support/checks/build.sh) the Java 8 and 11 versions of the jars, and saves the java 8 version for some of the subsequent jobs. + +#### basic job +Runs a subset of the following subjobs depending on what was selected by build-info +- author: [Verifies](../hadoop-ozone/dev-support/checks/author.sh) none of the Java files contain the @author annotation +- bats: [Checks](../hadoop-ozone/dev-support/checks/bats.sh) bash scripts, (using the [Bash Automated Testing System](https://github.com/bats-core/bats-core#bats-core-bash-automated-testing-system-2018)) +- checkstyle: [Runs](../hadoop-ozone/dev-support/checks/checkstyle.sh) 'mvn checkstyle' plugin to confirm Java source abides by Ozone coding conventions +- docs: [Builds](../hadoop-ozone/dev-support/checks/docs.sh) website with [Hugo](https://gohugo.io/) +- findbugs: [Runs](../hadoop-ozone/dev-support/checks/findbugs.sh) spotbugs static analysis on bytecode +- rat (release audit tool): [Confirms](../hadoop-ozone/dev-support/checks/rat.sh) source files include licenses +- unit: [Runs](../hadoop-ozone/dev-support/checks/unit.sh) 'mvn test' for all non integration tests + +#### dependency job +[Confirms](../hadoop-ozone/dev-support/checks/dependency.sh) hadoop-ozone/dist/src/main/license/bin/LICENSE.txt is up to date, (includes references to all the latest jar files). + +#### acceptance job +[Runs](../hadoop-ozone/dev-support/checks/acceptance.sh) smoketests using robot framework and a real docker compose cluster + +#### kubernetes job +[Runs](../hadoop-ozone/dev-support/checks/kubernetes.sh) k8s tests + +#### integration job +[Runs](../hadoop-ozone/dev-support/checks/integration.sh) 'mvn test' for all integration/minicluster tests, split into 3 subjobs: +- client +- filesystem-hdds +- ozone + +#### coverage job +[Merges](../hadoop-ozone/dev-support/checks/coverage.sh) the coverage data from the following jobs that were run earlier: +- acceptance +- basic +- integration + +### Cancelling Workflow +[This](./workflows/cancel-ci.yaml) workflow is triggered each time a [build-branch](ci.md#build-branch-workflow) workflow is triggered. It monitors all the jobs triggered by that workflow run for failure and cancels any continuing jobs after one fails. This reduces our GA usage. + +### close-prs Workflow +[This](./workflows/close-pending.yaml) workflow is scheduled each night at midnight; it closes PR's that have not been updated in the last 21 days, while letting the author know they are free to reopen. + +### comment-commands Workflow +[This](./workflows/comments.yaml) workflow is triggered each time a comment is added/edited to a PR. It checks to see if the body of the comment begins with one of the following strings and, if so, invokes the corresponding command. +- /close : [Close](./comment-commands/close.sh) pending pull request (with message saying author is free to reopen.) +- /help : [Show](./comment-commands/help.sh) all the available comment commands +- /label : [Add](./comment-commands/label.sh) new label to the issue: /label "label" +- /pending : [Add](./comment-commands/pending.sh) a REQUESTED_CHANGE type review to mark issue non-mergeable: /pending "reason" +- /ready : [Dismiss](./comment-commands/ready.sh) all the blocking reviews +- /retest : [Provide](./comment-commands/retest.sh) help on how to trigger new CI build + + +## Old/Deprecated Workflows +The following workflows no longer run but still exist on the [actions](https://github.com/apache/ozone/actions) page for historical reasons: +- [Build](https://github.com/apache/ozone/actions/workflows/main.yml) +- [build-branch](https://github.com/apache/ozone/actions/workflows/chaos.yml) +- [pr-check](https://github.com/apache/ozone/actions/workflows/pr.yml) + +Note that the deprecated [build-branch](https://github.com/apache/ozone/actions/workflows/chaos.yml) has the same name as the current [build-branch](https://github.com/apache/ozone/actions/workflows/post-commit.yml). (They can be distinguished by the URL.) + + +## Tips + +- When a build of the Ozone master branch fails, it's artifacts are stored [here](https://elek.github.io/ozone-build-results/). Review comment: Thanks @adoroszlai! (I'm embarassed to see I still get the possessive mixed up with contractions.) Good thing I have you to check my English! (Can you help me with my Magyar too?) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
