adoroszlai commented on a change in pull request #2607: URL: https://github.com/apache/ozone/pull/2607#discussion_r707252542
########## File path: .github/ci.md ########## @@ -0,0 +1,94 @@ +# Ozone CI with Github Actions + +The Ozone project uses Github Actions, (GA), for its CI system. GA are implemented with workflows, which are groups of *jobs* combined to accomplish a CI task, all defined in a single yaml file. The Ozone workflow yaml files are [here](./workflows). + +## Workflows + +### build-branch Workflow +This is the most important [workflow](./workflows/post-commit.yml). It runs the tests that verify the latest commits. + +It is triggered each time a pull request is created or synchronized, (ie when the remote branch is pushed to.) It is also "scheduled" on the master branch twice a day, (00:30 and 12:30). (Those are the runs [here](https://github.com/apache/ozone/actions/workflows/post-commit.yml?query=event%3Aschedule++) which are marked "scheduled", and have no branch label.) + +The build-branch workflow is divided into a number of different jobs, most of which run in parallel. Each of the jobs, (except "build-info" and "compile",) runs a subset of the test suite. They are all described below. + +#### build-info job + +[The build-info job script](../dev-support/ci/selective_ci_checks.sh) runs before the others and determines which of the other jobs are to be run. If the workflow was triggered by some event other than a PR, then all jobs/tests are run. They are also all run if the PR has a label containing the following string, "full tests needed". + +Otherwise, *build-info* first generates a list of files that were changed by the PR. It matches that list against a series of regex's, each of which is associated with a different job. It sets the appropriate flag for each match. Those boolean flags are used later in the run to decide whether the corresponding job should be run + +For example, a regex like the following is used to determine if the Kubernetes flag should be set. +``` + local pattern_array=( + "^hadoop-ozone/dev-support/checks/kubernetes.sh" + "^hadoop-ozone/dist/src/main/k8s" + ) +``` + + + +#### compile job +[Builds](../hadoop-ozone/dev-support/checks/build.sh) the Java 8 and 11 versions of the jars, and saves the java 8 version for some of the subsequent jobs. + +#### basic job +Runs a subset of the following subjobs depending on what was selected by build-info +- author: [Verifies](../hadoop-ozone/dev-support/checks/author.sh) none of the Java files contain the @author annotation +- bats: [Checks](../hadoop-ozone/dev-support/checks/bats.sh) bash scripts, (using the [Bash Automated Testing System](https://github.com/bats-core/bats-core#bats-core-bash-automated-testing-system-2018)) +- checkstyle: [Runs](../hadoop-ozone/dev-support/checks/checkstyle.sh) 'mvn checkstyle' plugin to confirm Java source abides by Ozone coding conventions +- docs: [Builds](../hadoop-ozone/dev-support/checks/docs.sh) website with [Hugo](https://gohugo.io/) +- findbugs: [Runs](../hadoop-ozone/dev-support/checks/findbugs.sh) spotbugs static analysis on bytecode +- rat (release audit tool): [Confirms](../hadoop-ozone/dev-support/checks/rat.sh) source files include licenses +- unit: [Runs](../hadoop-ozone/dev-support/checks/unit.sh) 'mvn test' for all non integration tests + +#### dependency job +[Confirms](../hadoop-ozone/dev-support/checks/dependency.sh) hadoop-ozone/dist/src/main/license/bin/LICENSE.txt is up to date, (includes references to all the latest jar files). + +#### acceptance job +[Runs](../hadoop-ozone/dev-support/checks/acceptance.sh) smoketests using robot framework and a real docker compose cluster + +#### kubernetes job +[Runs](../hadoop-ozone/dev-support/checks/kubernetes.sh) k8s tests + +#### integration job +[Runs](../hadoop-ozone/dev-support/checks/integration.sh) 'mvn test' for all integration/minicluster tests, split into 3 subjobs: +- client +- filesystem-hdds +- ozone Review comment: I would omit the list of specific sub-jobs. Sometimes we need to rebalance the test profiles to get similar run time. (BTW, a rebalance is probably overdue currently.) ########## File path: .github/ci.md ########## @@ -0,0 +1,94 @@ +# Ozone CI with Github Actions + +The Ozone project uses Github Actions, (GA), for its CI system. GA are implemented with workflows, which are groups of *jobs* combined to accomplish a CI task, all defined in a single yaml file. The Ozone workflow yaml files are [here](./workflows). + +## Workflows + +### build-branch Workflow +This is the most important [workflow](./workflows/post-commit.yml). It runs the tests that verify the latest commits. + +It is triggered each time a pull request is created or synchronized, (ie when the remote branch is pushed to.) It is also "scheduled" on the master branch twice a day, (00:30 and 12:30). (Those are the runs [here](https://github.com/apache/ozone/actions/workflows/post-commit.yml?query=event%3Aschedule++) which are marked "scheduled", and have no branch label.) + +The build-branch workflow is divided into a number of different jobs, most of which run in parallel. Each of the jobs, (except "build-info" and "compile",) runs a subset of the test suite. They are all described below. + +#### build-info job + +[The build-info job script](../dev-support/ci/selective_ci_checks.sh) runs before the others and determines which of the other jobs are to be run. If the workflow was triggered by some event other than a PR, then all jobs/tests are run. They are also all run if the PR has a label containing the following string, "full tests needed". + +Otherwise, *build-info* first generates a list of files that were changed by the PR. It matches that list against a series of regex's, each of which is associated with a different job. It sets the appropriate flag for each match. Those boolean flags are used later in the run to decide whether the corresponding job should be run + +For example, a regex like the following is used to determine if the Kubernetes flag should be set. +``` + local pattern_array=( + "^hadoop-ozone/dev-support/checks/kubernetes.sh" + "^hadoop-ozone/dist/src/main/k8s" + ) +``` + + + +#### compile job +[Builds](../hadoop-ozone/dev-support/checks/build.sh) the Java 8 and 11 versions of the jars, and saves the java 8 version for some of the subsequent jobs. + +#### basic job +Runs a subset of the following subjobs depending on what was selected by build-info +- author: [Verifies](../hadoop-ozone/dev-support/checks/author.sh) none of the Java files contain the @author annotation +- bats: [Checks](../hadoop-ozone/dev-support/checks/bats.sh) bash scripts, (using the [Bash Automated Testing System](https://github.com/bats-core/bats-core#bats-core-bash-automated-testing-system-2018)) +- checkstyle: [Runs](../hadoop-ozone/dev-support/checks/checkstyle.sh) 'mvn checkstyle' plugin to confirm Java source abides by Ozone coding conventions +- docs: [Builds](../hadoop-ozone/dev-support/checks/docs.sh) website with [Hugo](https://gohugo.io/) +- findbugs: [Runs](../hadoop-ozone/dev-support/checks/findbugs.sh) spotbugs static analysis on bytecode +- rat (release audit tool): [Confirms](../hadoop-ozone/dev-support/checks/rat.sh) source files include licenses +- unit: [Runs](../hadoop-ozone/dev-support/checks/unit.sh) 'mvn test' for all non integration tests + +#### dependency job +[Confirms](../hadoop-ozone/dev-support/checks/dependency.sh) hadoop-ozone/dist/src/main/license/bin/LICENSE.txt is up to date, (includes references to all the latest jar files). + +#### acceptance job +[Runs](../hadoop-ozone/dev-support/checks/acceptance.sh) smoketests using robot framework and a real docker compose cluster + +#### kubernetes job +[Runs](../hadoop-ozone/dev-support/checks/kubernetes.sh) k8s tests + +#### integration job +[Runs](../hadoop-ozone/dev-support/checks/integration.sh) 'mvn test' for all integration/minicluster tests, split into 3 subjobs: +- client +- filesystem-hdds +- ozone + +#### coverage job +[Merges](../hadoop-ozone/dev-support/checks/coverage.sh) the coverage data from the following jobs that were run earlier: +- acceptance +- basic +- integration + +### Cancelling Workflow +[This](./workflows/cancel-ci.yaml) workflow is triggered each time a [build-branch](ci.md#build-branch-workflow) workflow is triggered. It monitors all the jobs triggered by that workflow run for failure and cancels any continuing jobs after one fails. This reduces our GA usage. + +### close-prs Workflow +[This](./workflows/close-pending.yaml) workflow is scheduled each night at midnight; it closes PR's that have not been updated in the last 21 days, while letting the author know they are free to reopen. + +### comment-commands Workflow +[This](./workflows/comments.yaml) workflow is triggered each time a comment is added/edited to a PR. It checks to see if the body of the comment begins with one of the following strings and, if so, invokes the corresponding command. +- /close : [Close](./comment-commands/close.sh) pending pull request (with message saying author is free to reopen.) +- /help : [Show](./comment-commands/help.sh) all the available comment commands +- /label : [Add](./comment-commands/label.sh) new label to the issue: /label "label" +- /pending : [Add](./comment-commands/pending.sh) a REQUESTED_CHANGE type review to mark issue non-mergeable: /pending "reason" +- /ready : [Dismiss](./comment-commands/ready.sh) all the blocking reviews +- /retest : [Provide](./comment-commands/retest.sh) help on how to trigger new CI build + + +## Old/Deprecated Workflows +The following workflows no longer run but still exist on the [actions](https://github.com/apache/ozone/actions) page for historical reasons: +- [Build](https://github.com/apache/ozone/actions/workflows/main.yml) +- [build-branch](https://github.com/apache/ozone/actions/workflows/chaos.yml) +- [pr-check](https://github.com/apache/ozone/actions/workflows/pr.yml) + +Note that the deprecated [build-branch](https://github.com/apache/ozone/actions/workflows/chaos.yml) has the same name as the current [build-branch](https://github.com/apache/ozone/actions/workflows/post-commit.yml). (They can be distinguished by the URL.) + + +## Tips + +- When a build of the Ozone master branch fails, it's artifacts are stored [here](https://elek.github.io/ozone-build-results/). Review comment: ```suggestion - When a build of the Ozone master branch fails, its artifacts are stored [here](https://elek.github.io/ozone-build-results/). ``` ########## File path: .github/ci.md ########## @@ -0,0 +1,94 @@ +# Ozone CI with Github Actions + +The Ozone project uses Github Actions, (GA), for its CI system. GA are implemented with workflows, which are groups of *jobs* combined to accomplish a CI task, all defined in a single yaml file. The Ozone workflow yaml files are [here](./workflows). + +## Workflows + +### build-branch Workflow +This is the most important [workflow](./workflows/post-commit.yml). It runs the tests that verify the latest commits. + +It is triggered each time a pull request is created or synchronized, (ie when the remote branch is pushed to.) It is also "scheduled" on the master branch twice a day, (00:30 and 12:30). (Those are the runs [here](https://github.com/apache/ozone/actions/workflows/post-commit.yml?query=event%3Aschedule++) which are marked "scheduled", and have no branch label.) + +The build-branch workflow is divided into a number of different jobs, most of which run in parallel. Each of the jobs, (except "build-info" and "compile",) runs a subset of the test suite. They are all described below. + +#### build-info job + +[The build-info job script](../dev-support/ci/selective_ci_checks.sh) runs before the others and determines which of the other jobs are to be run. If the workflow was triggered by some event other than a PR, then all jobs/tests are run. They are also all run if the PR has a label containing the following string, "full tests needed". + +Otherwise, *build-info* first generates a list of files that were changed by the PR. It matches that list against a series of regex's, each of which is associated with a different job. It sets the appropriate flag for each match. Those boolean flags are used later in the run to decide whether the corresponding job should be run + +For example, a regex like the following is used to determine if the Kubernetes flag should be set. +``` + local pattern_array=( + "^hadoop-ozone/dev-support/checks/kubernetes.sh" + "^hadoop-ozone/dist/src/main/k8s" + ) +``` + + + +#### compile job +[Builds](../hadoop-ozone/dev-support/checks/build.sh) the Java 8 and 11 versions of the jars, and saves the java 8 version for some of the subsequent jobs. + +#### basic job +Runs a subset of the following subjobs depending on what was selected by build-info +- author: [Verifies](../hadoop-ozone/dev-support/checks/author.sh) none of the Java files contain the @author annotation +- bats: [Checks](../hadoop-ozone/dev-support/checks/bats.sh) bash scripts, (using the [Bash Automated Testing System](https://github.com/bats-core/bats-core#bats-core-bash-automated-testing-system-2018)) +- checkstyle: [Runs](../hadoop-ozone/dev-support/checks/checkstyle.sh) 'mvn checkstyle' plugin to confirm Java source abides by Ozone coding conventions +- docs: [Builds](../hadoop-ozone/dev-support/checks/docs.sh) website with [Hugo](https://gohugo.io/) +- findbugs: [Runs](../hadoop-ozone/dev-support/checks/findbugs.sh) spotbugs static analysis on bytecode +- rat (release audit tool): [Confirms](../hadoop-ozone/dev-support/checks/rat.sh) source files include licenses +- unit: [Runs](../hadoop-ozone/dev-support/checks/unit.sh) 'mvn test' for all non integration tests + +#### dependency job +[Confirms](../hadoop-ozone/dev-support/checks/dependency.sh) hadoop-ozone/dist/src/main/license/bin/LICENSE.txt is up to date, (includes references to all the latest jar files). + +#### acceptance job +[Runs](../hadoop-ozone/dev-support/checks/acceptance.sh) smoketests using robot framework and a real docker compose cluster + +#### kubernetes job +[Runs](../hadoop-ozone/dev-support/checks/kubernetes.sh) k8s tests + +#### integration job +[Runs](../hadoop-ozone/dev-support/checks/integration.sh) 'mvn test' for all integration/minicluster tests, split into 3 subjobs: +- client +- filesystem-hdds +- ozone + +#### coverage job +[Merges](../hadoop-ozone/dev-support/checks/coverage.sh) the coverage data from the following jobs that were run earlier: +- acceptance +- basic +- integration + +### Cancelling Workflow +[This](./workflows/cancel-ci.yaml) workflow is triggered each time a [build-branch](ci.md#build-branch-workflow) workflow is triggered. It monitors all the jobs triggered by that workflow run for failure and cancels any continuing jobs after one fails. This reduces our GA usage. Review comment: > monitors all the jobs triggered by that workflow run for failure and cancels any continuing jobs after one fails It doesn't really monitor the other run. It only checks for failure at the time the cancelling workflow is run. Any job that happens to fail after that time can only be caught by a subsequent run of the cancelling workflow. On the other hand, it checks status of all workflows running at that time, not only the one that triggered this cancelling workflow. ########## File path: .github/ci.md ########## @@ -0,0 +1,94 @@ +# Ozone CI with Github Actions + +The Ozone project uses Github Actions, (GA), for its CI system. GA are implemented with workflows, which are groups of *jobs* combined to accomplish a CI task, all defined in a single yaml file. The Ozone workflow yaml files are [here](./workflows). + +## Workflows + +### build-branch Workflow +This is the most important [workflow](./workflows/post-commit.yml). It runs the tests that verify the latest commits. + +It is triggered each time a pull request is created or synchronized, (ie when the remote branch is pushed to.) It is also "scheduled" on the master branch twice a day, (00:30 and 12:30). (Those are the runs [here](https://github.com/apache/ozone/actions/workflows/post-commit.yml?query=event%3Aschedule++) which are marked "scheduled", and have no branch label.) + +The build-branch workflow is divided into a number of different jobs, most of which run in parallel. Each of the jobs, (except "build-info" and "compile",) runs a subset of the test suite. They are all described below. + +#### build-info job + +[The build-info job script](../dev-support/ci/selective_ci_checks.sh) runs before the others and determines which of the other jobs are to be run. If the workflow was triggered by some event other than a PR, then all jobs/tests are run. They are also all run if the PR has a label containing the following string, "full tests needed". + +Otherwise, *build-info* first generates a list of files that were changed by the PR. It matches that list against a series of regex's, each of which is associated with a different job. It sets the appropriate flag for each match. Those boolean flags are used later in the run to decide whether the corresponding job should be run + +For example, a regex like the following is used to determine if the Kubernetes flag should be set. +``` + local pattern_array=( + "^hadoop-ozone/dev-support/checks/kubernetes.sh" + "^hadoop-ozone/dist/src/main/k8s" + ) +``` + + + +#### compile job +[Builds](../hadoop-ozone/dev-support/checks/build.sh) the Java 8 and 11 versions of the jars, and saves the java 8 version for some of the subsequent jobs. + +#### basic job +Runs a subset of the following subjobs depending on what was selected by build-info +- author: [Verifies](../hadoop-ozone/dev-support/checks/author.sh) none of the Java files contain the @author annotation +- bats: [Checks](../hadoop-ozone/dev-support/checks/bats.sh) bash scripts, (using the [Bash Automated Testing System](https://github.com/bats-core/bats-core#bats-core-bash-automated-testing-system-2018)) +- checkstyle: [Runs](../hadoop-ozone/dev-support/checks/checkstyle.sh) 'mvn checkstyle' plugin to confirm Java source abides by Ozone coding conventions +- docs: [Builds](../hadoop-ozone/dev-support/checks/docs.sh) website with [Hugo](https://gohugo.io/) +- findbugs: [Runs](../hadoop-ozone/dev-support/checks/findbugs.sh) spotbugs static analysis on bytecode +- rat (release audit tool): [Confirms](../hadoop-ozone/dev-support/checks/rat.sh) source files include licenses +- unit: [Runs](../hadoop-ozone/dev-support/checks/unit.sh) 'mvn test' for all non integration tests + +#### dependency job +[Confirms](../hadoop-ozone/dev-support/checks/dependency.sh) hadoop-ozone/dist/src/main/license/bin/LICENSE.txt is up to date, (includes references to all the latest jar files). Review comment: I don't think it really checks `LICENSE.txt`. Rather it looks for differences in the list of dependencies before and after the commit. Guards against unintentionally added dependencies, assuming if you take the time to update `jar-report.txt`, then you have also considered `LICENSE.txt`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
