adoroszlai commented on code in PR #382: URL: https://github.com/apache/ozone-site/pull/382#discussion_r3067643346
########## docs/08-developer-guide/03-test/04-continuous-integration.md: ########## @@ -1,15 +1,160 @@ --- -draft: true sidebar_label: Continuous Integration --- # Continuous Integration With GitHub Actions -**TODO:** File a subtask under [HDDS-9861](https://issues.apache.org/jira/browse/HDDS-9861) and complete this page or section. +If you are new to the project, **you do not need to understand every job** below on day one. The goal of this page is to help you get a green **`build-branch`** run on your fork, know where to look when something fails, and find deeper detail when you need it. -Aggregate content from our various GitHub actions guides, including +Apache Ozone uses [GitHub Actions](https://docs.github.com/en/actions) to build and test every meaningful change. Workflow files live in [`.github/workflows`](https://github.com/apache/ozone/tree/master/.github/workflows) in [`apache/ozone`](https://github.com/apache/ozone). A longer, file-by-file reference lives in [`.github/ci.md`](https://github.com/apache/ozone/blob/master/.github/ci.md). -- [ci.md](https://github.com/apache/ozone/blob/master/.github/ci.md) -- [CONTRIBUTING.md](https://github.com/apache/ozone/blob/master/CONTRIBUTING.md#check-your-contribution) -- https://cwiki.apache.org/confluence/display/OZONE/Ozone+CI+with+Github+Actions -- https://cwiki.apache.org/confluence/display/OZONE/Github+Actions+tips+and+tricks. +:::info Use the right repository + +This page is about **[`apache/ozone`](https://github.com/apache/ozone)** (the Ozone product source code). The documentation site you are reading comes from **[`apache/ozone-site`](https://github.com/apache/ozone-site)** and has its **own** CI. For website-only edits, use the [ozone-site contributing guide](https://github.com/apache/ozone-site/blob/master/CONTRIBUTING.md). + +::: + +## Start here: your first code contribution + +Follow these steps once; after that, pushing to your branch is the usual loop. + +1. **Fork and clone** [`apache/ozone`](https://github.com/apache/ozone) to **your** GitHub account, then clone **your fork** locally. You will push branches to `origin` on the fork, then open a PR to `apache/ozone`. +2. **Turn on Actions** on the fork so workflows actually run ([how to enable them](#enable-github-actions-on-your-fork)). +3. **Jira** — Create or choose an issue in [HDDS](https://issues.apache.org/jira/projects/HDDS/) (the Ozone Jira project; the name is historical). Need an account? Use the ASF [Jira self-service](https://selfserve.apache.org/jira-account.html?project=ozone) form. +4. **Branch** — Work on a branch, often named after the issue (for example `HDDS-1234`). +5. **Push** — When you push, GitHub should show a **`build-branch`** workflow run under the **Actions** tab on your fork. Wait for it to finish and fix any failures you can reproduce. +6. **Open the PR** — Use the [pull request template](https://github.com/apache/ozone/blob/master/.github/pull_request_template.md). When the change is ready for review, set the Jira to **Patch Available** so committers know to look. + +The full narrative (reviews, merging, Jira etiquette) is in the [Ozone contributing guide](https://github.com/apache/ozone/blob/master/CONTRIBUTING.md#contribute-your-modifications). + +:::tip You can lean on CI first + +Many contributors fix quick issues by reading the failing log on GitHub, then pushing a small follow-up commit. Running every check locally is **optional but helpful** for faster feedback; see [Run checks on your machine](#run-checks-on-your-machine). + +::: + +## Enable GitHub Actions on your fork + +New forks sometimes have workflows off until you allow them. + +1. Open **your fork** on GitHub → **Settings** → **Actions** → **General**. +2. Under **Actions permissions**, pick a policy that allows workflows to run (many people use **Allow all actions and reusable workflows** on personal forks). +3. Open the **Actions** tab. If GitHub asks to enable workflows, confirm so **`build-branch`** runs when you push. + +More detail: [Enabling or disabling GitHub Actions](https://docs.github.com/en/repositories/managing-your-repositorys-settings-and-features/enabling-or-disabling-github-actions-for-a-repository). + +## What you see in GitHub: `build-branch` + +Two names show up in docs; both mean “the main CI pipeline”: + +| What | Meaning | +| --- | --- | +| **`build-branch`** | The **name** of the workflow in the Actions tab. It comes from the `name:` field in [`post-commit.yml`](https://github.com/apache/ozone/blob/master/.github/workflows/post-commit.yml). | +| **`ci.yml`** | Where most **jobs** (compile, tests, and so on) are defined. `post-commit.yml` calls this file as a [reusable workflow](https://docs.github.com/en/actions/using-workflows/reusing-workflows). | + +So: **`post-commit.yml`** = front door; **`ci.yml`** = where the heavy lifting is described. + +When you push new commits to an open pull request, **newer runs can cancel older ones** still in progress ([concurrency](https://docs.github.com/en/actions/using-jobs/using-concurrency)). That is normal and saves time. + +## Run checks on your machine + +Running scripts locally catches problems before you push. You need a working dev environment first—see [Build with Maven](../../developer-guide/build/maven) and [Building from source](https://github.com/apache/ozone/blob/master/CONTRIBUTING.md#building-from-source) in `CONTRIBUTING.md`. + +From the **root of your clone** (the folder that contains `hadoop-ozone/`): + +```bash +./hadoop-ozone/dev-support/checks/build.sh +``` + +Most checks live in [`hadoop-ozone/dev-support/checks`](https://github.com/apache/ozone/tree/master/hadoop-ozone/dev-support/checks). The [Check your contribution](https://github.com/apache/ozone/blob/master/CONTRIBUTING.md#check-your-contribution) section groups them by rough duration: + +| Rough time | Scripts | What they do | +| --- | --- | --- | +| Build step | `build.sh` | Compile Ozone | +| Minutes | `author.sh`, `bats.sh`, `rat.sh`, `docs.sh`, `dependency.sh`, `checkstyle.sh`, `pmd.sh` | Style, license headers, docs, dependency list | +| ~10 minutes | `findbugs.sh`, `kubernetes.sh` | SpotBugs, small Kubernetes-related checks | +| An hour or more | `unit.sh`, `integration.sh`, `acceptance.sh` | Unit tests, mini-cluster tests, Docker Compose acceptance tests | + +More on test styles: [Acceptance tests](./acceptance-tests) on this site. For unit and integration testing (and running checks locally), see [Running Ozone smoke tests and unit tests](https://cwiki.apache.org/confluence/display/OZONE/Running+Ozone+Smoke+Tests+and+Unit+Tests) on the wiki until the dedicated **Unit tests** and **Integration tests** pages here are published. + +`integration.sh` and `acceptance.sh` can take extra arguments to run a subset; open the scripts to see options. Output usually lands under `target/` (for example `target/docs`). + +## Why did CI skip some jobs? + +Not every pull request runs every job. A step called **build-info** runs [`selective_ci_checks.sh`](https://github.com/apache/ozone/blob/master/dev-support/ci/selective_ci_checks.sh) and only enables jobs that match the files you changed—unless: + +- the run is **not** from a PR, or +- the PR has the **`full tests needed`** [label](https://docs.github.com/en/issues/using-labels-and-milestones-to-track-work/managing-labels). + +So a focused change might show fewer checks than a large refactor. **That is expected.** Reviewers can add **`full tests needed`** when the full matrix is required. If you think the wrong jobs were skipped, **ask on the PR**; reviewers are used to that question. + +## What the main CI jobs do (overview) + +The list below matches [`.github/ci.md`](https://github.com/apache/ozone/blob/master/.github/ci.md). Treat it as a map when reading logs, not something to memorize. + +- **build-info** — Decides which other jobs run (selective CI). +- **compile** — Builds with Java 8 and 11 via [`build.sh`](https://github.com/apache/ozone/blob/master/hadoop-ozone/dev-support/checks/build.sh); later jobs typically use the Java 8 build. +- **basic** — Checks like author tags, BATS, Checkstyle, Hugo for docs, SpotBugs, PMD, RAT—depending on what was selected. +- **unit** — [`unit.sh`](https://github.com/apache/ozone/blob/master/hadoop-ozone/dev-support/checks/unit.sh) and [`native.sh`](https://github.com/apache/ozone/blob/master/hadoop-ozone/dev-support/checks/native.sh) for RocksDB-native tests. +- **dependency** — Compares JARs to [`jar-report.txt`](https://github.com/apache/ozone/blob/master/hadoop-ozone/dist/src/main/license/jar-report.txt). +- **acceptance** — [`acceptance.sh`](https://github.com/apache/ozone/blob/master/hadoop-ozone/dev-support/checks/acceptance.sh) (Robot Framework + Docker Compose; variants like secure / unsecure / misc). +- **kubernetes** — [`kubernetes.sh`](https://github.com/apache/ozone/blob/master/hadoop-ozone/dev-support/checks/kubernetes.sh). +- **integration** — [`integration.sh`](https://github.com/apache/ozone/blob/master/hadoop-ozone/dev-support/checks/integration.sh) (mini-cluster style tests, often sharded in CI). +- **coverage** — [`coverage.sh`](https://github.com/apache/ozone/blob/master/hadoop-ozone/dev-support/checks/coverage.sh) merges coverage when earlier jobs produced it. + +Some jobs use a **matrix** (for example multiple Java versions) with **fail-fast**: if one matrix leg fails, the others in that matrix stop. Unrelated jobs can still run until they finish or fail. Review Comment: `fail-fast` behavior has been disabled after GitHub made it possible to re-run specific jobs (HDDS-6464). ########## docs/08-developer-guide/03-test/04-continuous-integration.md: ########## @@ -1,15 +1,160 @@ --- -draft: true sidebar_label: Continuous Integration --- # Continuous Integration With GitHub Actions -**TODO:** File a subtask under [HDDS-9861](https://issues.apache.org/jira/browse/HDDS-9861) and complete this page or section. +If you are new to the project, **you do not need to understand every job** below on day one. The goal of this page is to help you get a green **`build-branch`** run on your fork, know where to look when something fails, and find deeper detail when you need it. -Aggregate content from our various GitHub actions guides, including +Apache Ozone uses [GitHub Actions](https://docs.github.com/en/actions) to build and test every meaningful change. Workflow files live in [`.github/workflows`](https://github.com/apache/ozone/tree/master/.github/workflows) in [`apache/ozone`](https://github.com/apache/ozone). A longer, file-by-file reference lives in [`.github/ci.md`](https://github.com/apache/ozone/blob/master/.github/ci.md). -- [ci.md](https://github.com/apache/ozone/blob/master/.github/ci.md) -- [CONTRIBUTING.md](https://github.com/apache/ozone/blob/master/CONTRIBUTING.md#check-your-contribution) -- https://cwiki.apache.org/confluence/display/OZONE/Ozone+CI+with+Github+Actions -- https://cwiki.apache.org/confluence/display/OZONE/Github+Actions+tips+and+tricks. +:::info Use the right repository + +This page is about **[`apache/ozone`](https://github.com/apache/ozone)** (the Ozone product source code). The documentation site you are reading comes from **[`apache/ozone-site`](https://github.com/apache/ozone-site)** and has its **own** CI. For website-only edits, use the [ozone-site contributing guide](https://github.com/apache/ozone-site/blob/master/CONTRIBUTING.md). + +::: + +## Start here: your first code contribution + +Follow these steps once; after that, pushing to your branch is the usual loop. + +1. **Fork and clone** [`apache/ozone`](https://github.com/apache/ozone) to **your** GitHub account, then clone **your fork** locally. You will push branches to `origin` on the fork, then open a PR to `apache/ozone`. +2. **Turn on Actions** on the fork so workflows actually run ([how to enable them](#enable-github-actions-on-your-fork)). +3. **Jira** — Create or choose an issue in [HDDS](https://issues.apache.org/jira/projects/HDDS/) (the Ozone Jira project; the name is historical). Need an account? Use the ASF [Jira self-service](https://selfserve.apache.org/jira-account.html?project=ozone) form. +4. **Branch** — Work on a branch, often named after the issue (for example `HDDS-1234`). +5. **Push** — When you push, GitHub should show a **`build-branch`** workflow run under the **Actions** tab on your fork. Wait for it to finish and fix any failures you can reproduce. +6. **Open the PR** — Use the [pull request template](https://github.com/apache/ozone/blob/master/.github/pull_request_template.md). When the change is ready for review, set the Jira to **Patch Available** so committers know to look. + +The full narrative (reviews, merging, Jira etiquette) is in the [Ozone contributing guide](https://github.com/apache/ozone/blob/master/CONTRIBUTING.md#contribute-your-modifications). + +:::tip You can lean on CI first + +Many contributors fix quick issues by reading the failing log on GitHub, then pushing a small follow-up commit. Running every check locally is **optional but helpful** for faster feedback; see [Run checks on your machine](#run-checks-on-your-machine). + +::: + +## Enable GitHub Actions on your fork + +New forks sometimes have workflows off until you allow them. + +1. Open **your fork** on GitHub → **Settings** → **Actions** → **General**. +2. Under **Actions permissions**, pick a policy that allows workflows to run (many people use **Allow all actions and reusable workflows** on personal forks). +3. Open the **Actions** tab. If GitHub asks to enable workflows, confirm so **`build-branch`** runs when you push. + +More detail: [Enabling or disabling GitHub Actions](https://docs.github.com/en/repositories/managing-your-repositorys-settings-and-features/enabling-or-disabling-github-actions-for-a-repository). + +## What you see in GitHub: `build-branch` + +Two names show up in docs; both mean “the main CI pipeline”: + +| What | Meaning | +| --- | --- | +| **`build-branch`** | The **name** of the workflow in the Actions tab. It comes from the `name:` field in [`post-commit.yml`](https://github.com/apache/ozone/blob/master/.github/workflows/post-commit.yml). | +| **`ci.yml`** | Where most **jobs** (compile, tests, and so on) are defined. `post-commit.yml` calls this file as a [reusable workflow](https://docs.github.com/en/actions/using-workflows/reusing-workflows). | + +So: **`post-commit.yml`** = front door; **`ci.yml`** = where the heavy lifting is described. + +When you push new commits to an open pull request, **newer runs can cancel older ones** still in progress ([concurrency](https://docs.github.com/en/actions/using-jobs/using-concurrency)). That is normal and saves time. + +## Run checks on your machine + +Running scripts locally catches problems before you push. You need a working dev environment first—see [Build with Maven](../../developer-guide/build/maven) and [Building from source](https://github.com/apache/ozone/blob/master/CONTRIBUTING.md#building-from-source) in `CONTRIBUTING.md`. + +From the **root of your clone** (the folder that contains `hadoop-ozone/`): + +```bash +./hadoop-ozone/dev-support/checks/build.sh +``` + +Most checks live in [`hadoop-ozone/dev-support/checks`](https://github.com/apache/ozone/tree/master/hadoop-ozone/dev-support/checks). The [Check your contribution](https://github.com/apache/ozone/blob/master/CONTRIBUTING.md#check-your-contribution) section groups them by rough duration: + +| Rough time | Scripts | What they do | +| --- | --- | --- | +| Build step | `build.sh` | Compile Ozone | +| Minutes | `author.sh`, `bats.sh`, `rat.sh`, `docs.sh`, `dependency.sh`, `checkstyle.sh`, `pmd.sh` | Style, license headers, docs, dependency list | +| ~10 minutes | `findbugs.sh`, `kubernetes.sh` | SpotBugs, small Kubernetes-related checks | +| An hour or more | `unit.sh`, `integration.sh`, `acceptance.sh` | Unit tests, mini-cluster tests, Docker Compose acceptance tests | + +More on test styles: [Acceptance tests](./acceptance-tests) on this site. For unit and integration testing (and running checks locally), see [Running Ozone smoke tests and unit tests](https://cwiki.apache.org/confluence/display/OZONE/Running+Ozone+Smoke+Tests+and+Unit+Tests) on the wiki until the dedicated **Unit tests** and **Integration tests** pages here are published. + +`integration.sh` and `acceptance.sh` can take extra arguments to run a subset; open the scripts to see options. Output usually lands under `target/` (for example `target/docs`). + +## Why did CI skip some jobs? + +Not every pull request runs every job. A step called **build-info** runs [`selective_ci_checks.sh`](https://github.com/apache/ozone/blob/master/dev-support/ci/selective_ci_checks.sh) and only enables jobs that match the files you changed—unless: + +- the run is **not** from a PR, or +- the PR has the **`full tests needed`** [label](https://docs.github.com/en/issues/using-labels-and-milestones-to-track-work/managing-labels). + +So a focused change might show fewer checks than a large refactor. **That is expected.** Reviewers can add **`full tests needed`** when the full matrix is required. If you think the wrong jobs were skipped, **ask on the PR**; reviewers are used to that question. + +## What the main CI jobs do (overview) + +The list below matches [`.github/ci.md`](https://github.com/apache/ozone/blob/master/.github/ci.md). Treat it as a map when reading logs, not something to memorize. + +- **build-info** — Decides which other jobs run (selective CI). +- **compile** — Builds with Java 8 and 11 via [`build.sh`](https://github.com/apache/ozone/blob/master/hadoop-ozone/dev-support/checks/build.sh); later jobs typically use the Java 8 build. Review Comment: - `build` performs a full build, its output is used by later jobs. - `compile` re-builds with various Java versions. Also, it uses the source tarball (i.e. release artifact) produced by `build`, not the git repo. ########## docs/08-developer-guide/03-test/04-continuous-integration.md: ########## @@ -1,15 +1,160 @@ --- -draft: true sidebar_label: Continuous Integration --- # Continuous Integration With GitHub Actions -**TODO:** File a subtask under [HDDS-9861](https://issues.apache.org/jira/browse/HDDS-9861) and complete this page or section. +If you are new to the project, **you do not need to understand every job** below on day one. The goal of this page is to help you get a green **`build-branch`** run on your fork, know where to look when something fails, and find deeper detail when you need it. -Aggregate content from our various GitHub actions guides, including +Apache Ozone uses [GitHub Actions](https://docs.github.com/en/actions) to build and test every meaningful change. Workflow files live in [`.github/workflows`](https://github.com/apache/ozone/tree/master/.github/workflows) in [`apache/ozone`](https://github.com/apache/ozone). A longer, file-by-file reference lives in [`.github/ci.md`](https://github.com/apache/ozone/blob/master/.github/ci.md). -- [ci.md](https://github.com/apache/ozone/blob/master/.github/ci.md) -- [CONTRIBUTING.md](https://github.com/apache/ozone/blob/master/CONTRIBUTING.md#check-your-contribution) -- https://cwiki.apache.org/confluence/display/OZONE/Ozone+CI+with+Github+Actions -- https://cwiki.apache.org/confluence/display/OZONE/Github+Actions+tips+and+tricks. +:::info Use the right repository + +This page is about **[`apache/ozone`](https://github.com/apache/ozone)** (the Ozone product source code). The documentation site you are reading comes from **[`apache/ozone-site`](https://github.com/apache/ozone-site)** and has its **own** CI. For website-only edits, use the [ozone-site contributing guide](https://github.com/apache/ozone-site/blob/master/CONTRIBUTING.md). + +::: + +## Start here: your first code contribution + +Follow these steps once; after that, pushing to your branch is the usual loop. + +1. **Fork and clone** [`apache/ozone`](https://github.com/apache/ozone) to **your** GitHub account, then clone **your fork** locally. You will push branches to `origin` on the fork, then open a PR to `apache/ozone`. +2. **Turn on Actions** on the fork so workflows actually run ([how to enable them](#enable-github-actions-on-your-fork)). +3. **Jira** — Create or choose an issue in [HDDS](https://issues.apache.org/jira/projects/HDDS/) (the Ozone Jira project; the name is historical). Need an account? Use the ASF [Jira self-service](https://selfserve.apache.org/jira-account.html?project=ozone) form. +4. **Branch** — Work on a branch, often named after the issue (for example `HDDS-1234`). +5. **Push** — When you push, GitHub should show a **`build-branch`** workflow run under the **Actions** tab on your fork. Wait for it to finish and fix any failures you can reproduce. +6. **Open the PR** — Use the [pull request template](https://github.com/apache/ozone/blob/master/.github/pull_request_template.md). When the change is ready for review, set the Jira to **Patch Available** so committers know to look. + +The full narrative (reviews, merging, Jira etiquette) is in the [Ozone contributing guide](https://github.com/apache/ozone/blob/master/CONTRIBUTING.md#contribute-your-modifications). + +:::tip You can lean on CI first + +Many contributors fix quick issues by reading the failing log on GitHub, then pushing a small follow-up commit. Running every check locally is **optional but helpful** for faster feedback; see [Run checks on your machine](#run-checks-on-your-machine). + +::: + +## Enable GitHub Actions on your fork + +New forks sometimes have workflows off until you allow them. + +1. Open **your fork** on GitHub → **Settings** → **Actions** → **General**. +2. Under **Actions permissions**, pick a policy that allows workflows to run (many people use **Allow all actions and reusable workflows** on personal forks). +3. Open the **Actions** tab. If GitHub asks to enable workflows, confirm so **`build-branch`** runs when you push. + +More detail: [Enabling or disabling GitHub Actions](https://docs.github.com/en/repositories/managing-your-repositorys-settings-and-features/enabling-or-disabling-github-actions-for-a-repository). + +## What you see in GitHub: `build-branch` + +Two names show up in docs; both mean “the main CI pipeline”: + +| What | Meaning | +| --- | --- | +| **`build-branch`** | The **name** of the workflow in the Actions tab. It comes from the `name:` field in [`post-commit.yml`](https://github.com/apache/ozone/blob/master/.github/workflows/post-commit.yml). | +| **`ci.yml`** | Where most **jobs** (compile, tests, and so on) are defined. `post-commit.yml` calls this file as a [reusable workflow](https://docs.github.com/en/actions/using-workflows/reusing-workflows). | + +So: **`post-commit.yml`** = front door; **`ci.yml`** = where the heavy lifting is described. + +When you push new commits to an open pull request, **newer runs can cancel older ones** still in progress ([concurrency](https://docs.github.com/en/actions/using-jobs/using-concurrency)). That is normal and saves time. + +## Run checks on your machine + +Running scripts locally catches problems before you push. You need a working dev environment first—see [Build with Maven](../../developer-guide/build/maven) and [Building from source](https://github.com/apache/ozone/blob/master/CONTRIBUTING.md#building-from-source) in `CONTRIBUTING.md`. + +From the **root of your clone** (the folder that contains `hadoop-ozone/`): + +```bash +./hadoop-ozone/dev-support/checks/build.sh +``` + +Most checks live in [`hadoop-ozone/dev-support/checks`](https://github.com/apache/ozone/tree/master/hadoop-ozone/dev-support/checks). The [Check your contribution](https://github.com/apache/ozone/blob/master/CONTRIBUTING.md#check-your-contribution) section groups them by rough duration: + +| Rough time | Scripts | What they do | +| --- | --- | --- | +| Build step | `build.sh` | Compile Ozone | +| Minutes | `author.sh`, `bats.sh`, `rat.sh`, `docs.sh`, `dependency.sh`, `checkstyle.sh`, `pmd.sh` | Style, license headers, docs, dependency list | +| ~10 minutes | `findbugs.sh`, `kubernetes.sh` | SpotBugs, small Kubernetes-related checks | +| An hour or more | `unit.sh`, `integration.sh`, `acceptance.sh` | Unit tests, mini-cluster tests, Docker Compose acceptance tests | + +More on test styles: [Acceptance tests](./acceptance-tests) on this site. For unit and integration testing (and running checks locally), see [Running Ozone smoke tests and unit tests](https://cwiki.apache.org/confluence/display/OZONE/Running+Ozone+Smoke+Tests+and+Unit+Tests) on the wiki until the dedicated **Unit tests** and **Integration tests** pages here are published. + +`integration.sh` and `acceptance.sh` can take extra arguments to run a subset; open the scripts to see options. Output usually lands under `target/` (for example `target/docs`). + +## Why did CI skip some jobs? + +Not every pull request runs every job. A step called **build-info** runs [`selective_ci_checks.sh`](https://github.com/apache/ozone/blob/master/dev-support/ci/selective_ci_checks.sh) and only enables jobs that match the files you changed—unless: + +- the run is **not** from a PR, or +- the PR has the **`full tests needed`** [label](https://docs.github.com/en/issues/using-labels-and-milestones-to-track-work/managing-labels). + +So a focused change might show fewer checks than a large refactor. **That is expected.** Reviewers can add **`full tests needed`** when the full matrix is required. If you think the wrong jobs were skipped, **ask on the PR**; reviewers are used to that question. + +## What the main CI jobs do (overview) + +The list below matches [`.github/ci.md`](https://github.com/apache/ozone/blob/master/.github/ci.md). Treat it as a map when reading logs, not something to memorize. + +- **build-info** — Decides which other jobs run (selective CI). +- **compile** — Builds with Java 8 and 11 via [`build.sh`](https://github.com/apache/ozone/blob/master/hadoop-ozone/dev-support/checks/build.sh); later jobs typically use the Java 8 build. +- **basic** — Checks like author tags, BATS, Checkstyle, Hugo for docs, SpotBugs, PMD, RAT—depending on what was selected. +- **unit** — [`unit.sh`](https://github.com/apache/ozone/blob/master/hadoop-ozone/dev-support/checks/unit.sh) and [`native.sh`](https://github.com/apache/ozone/blob/master/hadoop-ozone/dev-support/checks/native.sh) for RocksDB-native tests. +- **dependency** — Compares JARs to [`jar-report.txt`](https://github.com/apache/ozone/blob/master/hadoop-ozone/dist/src/main/license/jar-report.txt). +- **acceptance** — [`acceptance.sh`](https://github.com/apache/ozone/blob/master/hadoop-ozone/dev-support/checks/acceptance.sh) (Robot Framework + Docker Compose; variants like secure / unsecure / misc). +- **kubernetes** — [`kubernetes.sh`](https://github.com/apache/ozone/blob/master/hadoop-ozone/dev-support/checks/kubernetes.sh). +- **integration** — [`integration.sh`](https://github.com/apache/ozone/blob/master/hadoop-ozone/dev-support/checks/integration.sh) (mini-cluster style tests, often sharded in CI). +- **coverage** — [`coverage.sh`](https://github.com/apache/ozone/blob/master/hadoop-ozone/dev-support/checks/coverage.sh) merges coverage when earlier jobs produced it. + +Some jobs use a **matrix** (for example multiple Java versions) with **fail-fast**: if one matrix leg fails, the others in that matrix stop. Unrelated jobs can still run until they finish or fail. + +## Other workflows + +The [workflows directory](https://github.com/apache/ozone/tree/master/.github/workflows) also contains jobs for caches, labeling, Ratis builds, repeating tests, generated docs, and more. The folder on `master` is the up-to-date list. + +### Stale pull requests + +[`close-stale-prs.yaml`](https://github.com/apache/ozone/blob/master/.github/workflows/close-stale-prs.yaml) runs on a timer and uses [actions/stale](https://github.com/actions/stale) to nudge and eventually close very inactive PRs. Exact timings are in that file. + +## If something fails + +:::note Green CI is a team norm + +A red check does not mean you did something wrong—it means the run found something to fix. It happens to everyone. + +::: + +1. Open the failed **`build-branch`** run → click the red job → read the **log** from the bottom upward for the first error. +2. If the job uploaded **Artifacts**, download them from the run summary (they expire after a short time). +3. Try the same **check script** locally if you have the environment set up ([Run checks on your machine](#run-checks-on-your-machine)). +4. To re-run without new code, use **Re-run all jobs** or **Re-run failed jobs** on the run page, or: + +```bash +git commit --allow-empty -m 'trigger new CI check' +``` Review Comment: I think we should omit both "Re-run all jobs" and the empty commit. They are not optimal ways to trigger new run. Committers: re-run failed jobs. Other contributors: wait for some committer to do so; ask for it if it does not happen within reasonable time (which depends on time of day, day of week, holidays, etc.). Checks that need any of the artifacts with 1-day expiry should be re-run within that period. In any case, this should be done only for transient errors / flaky tests. ########## docs/08-developer-guide/03-test/04-continuous-integration.md: ########## @@ -1,15 +1,160 @@ --- -draft: true sidebar_label: Continuous Integration --- # Continuous Integration With GitHub Actions -**TODO:** File a subtask under [HDDS-9861](https://issues.apache.org/jira/browse/HDDS-9861) and complete this page or section. +If you are new to the project, **you do not need to understand every job** below on day one. The goal of this page is to help you get a green **`build-branch`** run on your fork, know where to look when something fails, and find deeper detail when you need it. -Aggregate content from our various GitHub actions guides, including +Apache Ozone uses [GitHub Actions](https://docs.github.com/en/actions) to build and test every meaningful change. Workflow files live in [`.github/workflows`](https://github.com/apache/ozone/tree/master/.github/workflows) in [`apache/ozone`](https://github.com/apache/ozone). A longer, file-by-file reference lives in [`.github/ci.md`](https://github.com/apache/ozone/blob/master/.github/ci.md). -- [ci.md](https://github.com/apache/ozone/blob/master/.github/ci.md) -- [CONTRIBUTING.md](https://github.com/apache/ozone/blob/master/CONTRIBUTING.md#check-your-contribution) -- https://cwiki.apache.org/confluence/display/OZONE/Ozone+CI+with+Github+Actions -- https://cwiki.apache.org/confluence/display/OZONE/Github+Actions+tips+and+tricks. +:::info Use the right repository + +This page is about **[`apache/ozone`](https://github.com/apache/ozone)** (the Ozone product source code). The documentation site you are reading comes from **[`apache/ozone-site`](https://github.com/apache/ozone-site)** and has its **own** CI. For website-only edits, use the [ozone-site contributing guide](https://github.com/apache/ozone-site/blob/master/CONTRIBUTING.md). + +::: + +## Start here: your first code contribution + +Follow these steps once; after that, pushing to your branch is the usual loop. + +1. **Fork and clone** [`apache/ozone`](https://github.com/apache/ozone) to **your** GitHub account, then clone **your fork** locally. You will push branches to `origin` on the fork, then open a PR to `apache/ozone`. +2. **Turn on Actions** on the fork so workflows actually run ([how to enable them](#enable-github-actions-on-your-fork)). +3. **Jira** — Create or choose an issue in [HDDS](https://issues.apache.org/jira/projects/HDDS/) (the Ozone Jira project; the name is historical). Need an account? Use the ASF [Jira self-service](https://selfserve.apache.org/jira-account.html?project=ozone) form. +4. **Branch** — Work on a branch, often named after the issue (for example `HDDS-1234`). +5. **Push** — When you push, GitHub should show a **`build-branch`** workflow run under the **Actions** tab on your fork. Wait for it to finish and fix any failures you can reproduce. +6. **Open the PR** — Use the [pull request template](https://github.com/apache/ozone/blob/master/.github/pull_request_template.md). When the change is ready for review, set the Jira to **Patch Available** so committers know to look. + +The full narrative (reviews, merging, Jira etiquette) is in the [Ozone contributing guide](https://github.com/apache/ozone/blob/master/CONTRIBUTING.md#contribute-your-modifications). + +:::tip You can lean on CI first + +Many contributors fix quick issues by reading the failing log on GitHub, then pushing a small follow-up commit. Running every check locally is **optional but helpful** for faster feedback; see [Run checks on your machine](#run-checks-on-your-machine). + +::: + +## Enable GitHub Actions on your fork + +New forks sometimes have workflows off until you allow them. + +1. Open **your fork** on GitHub → **Settings** → **Actions** → **General**. +2. Under **Actions permissions**, pick a policy that allows workflows to run (many people use **Allow all actions and reusable workflows** on personal forks). +3. Open the **Actions** tab. If GitHub asks to enable workflows, confirm so **`build-branch`** runs when you push. + +More detail: [Enabling or disabling GitHub Actions](https://docs.github.com/en/repositories/managing-your-repositorys-settings-and-features/enabling-or-disabling-github-actions-for-a-repository). + +## What you see in GitHub: `build-branch` + +Two names show up in docs; both mean “the main CI pipeline”: + +| What | Meaning | +| --- | --- | +| **`build-branch`** | The **name** of the workflow in the Actions tab. It comes from the `name:` field in [`post-commit.yml`](https://github.com/apache/ozone/blob/master/.github/workflows/post-commit.yml). | +| **`ci.yml`** | Where most **jobs** (compile, tests, and so on) are defined. `post-commit.yml` calls this file as a [reusable workflow](https://docs.github.com/en/actions/using-workflows/reusing-workflows). | + +So: **`post-commit.yml`** = front door; **`ci.yml`** = where the heavy lifting is described. + +When you push new commits to an open pull request, **newer runs can cancel older ones** still in progress ([concurrency](https://docs.github.com/en/actions/using-jobs/using-concurrency)). That is normal and saves time. + +## Run checks on your machine + +Running scripts locally catches problems before you push. You need a working dev environment first—see [Build with Maven](../../developer-guide/build/maven) and [Building from source](https://github.com/apache/ozone/blob/master/CONTRIBUTING.md#building-from-source) in `CONTRIBUTING.md`. + +From the **root of your clone** (the folder that contains `hadoop-ozone/`): + +```bash +./hadoop-ozone/dev-support/checks/build.sh +``` + +Most checks live in [`hadoop-ozone/dev-support/checks`](https://github.com/apache/ozone/tree/master/hadoop-ozone/dev-support/checks). The [Check your contribution](https://github.com/apache/ozone/blob/master/CONTRIBUTING.md#check-your-contribution) section groups them by rough duration: + +| Rough time | Scripts | What they do | +| --- | --- | --- | +| Build step | `build.sh` | Compile Ozone | +| Minutes | `author.sh`, `bats.sh`, `rat.sh`, `docs.sh`, `dependency.sh`, `checkstyle.sh`, `pmd.sh` | Style, license headers, docs, dependency list | +| ~10 minutes | `findbugs.sh`, `kubernetes.sh` | SpotBugs, small Kubernetes-related checks | +| An hour or more | `unit.sh`, `integration.sh`, `acceptance.sh` | Unit tests, mini-cluster tests, Docker Compose acceptance tests | + +More on test styles: [Acceptance tests](./acceptance-tests) on this site. For unit and integration testing (and running checks locally), see [Running Ozone smoke tests and unit tests](https://cwiki.apache.org/confluence/display/OZONE/Running+Ozone+Smoke+Tests+and+Unit+Tests) on the wiki until the dedicated **Unit tests** and **Integration tests** pages here are published. + +`integration.sh` and `acceptance.sh` can take extra arguments to run a subset; open the scripts to see options. Output usually lands under `target/` (for example `target/docs`). + +## Why did CI skip some jobs? + +Not every pull request runs every job. A step called **build-info** runs [`selective_ci_checks.sh`](https://github.com/apache/ozone/blob/master/dev-support/ci/selective_ci_checks.sh) and only enables jobs that match the files you changed—unless: + +- the run is **not** from a PR, or +- the PR has the **`full tests needed`** [label](https://docs.github.com/en/issues/using-labels-and-milestones-to-track-work/managing-labels). + +So a focused change might show fewer checks than a large refactor. **That is expected.** Reviewers can add **`full tests needed`** when the full matrix is required. If you think the wrong jobs were skipped, **ask on the PR**; reviewers are used to that question. + +## What the main CI jobs do (overview) + +The list below matches [`.github/ci.md`](https://github.com/apache/ozone/blob/master/.github/ci.md). Treat it as a map when reading logs, not something to memorize. + +- **build-info** — Decides which other jobs run (selective CI). +- **compile** — Builds with Java 8 and 11 via [`build.sh`](https://github.com/apache/ozone/blob/master/hadoop-ozone/dev-support/checks/build.sh); later jobs typically use the Java 8 build. +- **basic** — Checks like author tags, BATS, Checkstyle, Hugo for docs, SpotBugs, PMD, RAT—depending on what was selected. +- **unit** — [`unit.sh`](https://github.com/apache/ozone/blob/master/hadoop-ozone/dev-support/checks/unit.sh) and [`native.sh`](https://github.com/apache/ozone/blob/master/hadoop-ozone/dev-support/checks/native.sh) for RocksDB-native tests. +- **dependency** — Compares JARs to [`jar-report.txt`](https://github.com/apache/ozone/blob/master/hadoop-ozone/dist/src/main/license/jar-report.txt). +- **acceptance** — [`acceptance.sh`](https://github.com/apache/ozone/blob/master/hadoop-ozone/dev-support/checks/acceptance.sh) (Robot Framework + Docker Compose; variants like secure / unsecure / misc). +- **kubernetes** — [`kubernetes.sh`](https://github.com/apache/ozone/blob/master/hadoop-ozone/dev-support/checks/kubernetes.sh). +- **integration** — [`integration.sh`](https://github.com/apache/ozone/blob/master/hadoop-ozone/dev-support/checks/integration.sh) (mini-cluster style tests, often sharded in CI). +- **coverage** — [`coverage.sh`](https://github.com/apache/ozone/blob/master/hadoop-ozone/dev-support/checks/coverage.sh) merges coverage when earlier jobs produced it. + +Some jobs use a **matrix** (for example multiple Java versions) with **fail-fast**: if one matrix leg fails, the others in that matrix stop. Unrelated jobs can still run until they finish or fail. + +## Other workflows + +The [workflows directory](https://github.com/apache/ozone/tree/master/.github/workflows) also contains jobs for caches, labeling, Ratis builds, repeating tests, generated docs, and more. The folder on `master` is the up-to-date list. + +### Stale pull requests + +[`close-stale-prs.yaml`](https://github.com/apache/ozone/blob/master/.github/workflows/close-stale-prs.yaml) runs on a timer and uses [actions/stale](https://github.com/actions/stale) to nudge and eventually close very inactive PRs. Exact timings are in that file. + +## If something fails + +:::note Green CI is a team norm + +A red check does not mean you did something wrong—it means the run found something to fix. It happens to everyone. + +::: + +1. Open the failed **`build-branch`** run → click the red job → read the **log** from the bottom upward for the first error. +2. If the job uploaded **Artifacts**, download them from the run summary (they expire after a short time). +3. Try the same **check script** locally if you have the environment set up ([Run checks on your machine](#run-checks-on-your-machine)). +4. To re-run without new code, use **Re-run all jobs** or **Re-run failed jobs** on the run page, or: + +```bash +git commit --allow-empty -m 'trigger new CI check' +``` + +Failures on the **main** repo’s default branch sometimes leave extra artifacts on [ozone build results](https://elek.github.io/ozone-build-results/) (community mirror). Review Comment: Marton's build result archive is no longer updated. Forked one is available at https://github.com/adoroszlai/ozone-build-results/ ########## docs/08-developer-guide/03-test/04-continuous-integration.md: ########## @@ -1,15 +1,160 @@ --- -draft: true sidebar_label: Continuous Integration --- # Continuous Integration With GitHub Actions -**TODO:** File a subtask under [HDDS-9861](https://issues.apache.org/jira/browse/HDDS-9861) and complete this page or section. +If you are new to the project, **you do not need to understand every job** below on day one. The goal of this page is to help you get a green **`build-branch`** run on your fork, know where to look when something fails, and find deeper detail when you need it. -Aggregate content from our various GitHub actions guides, including +Apache Ozone uses [GitHub Actions](https://docs.github.com/en/actions) to build and test every meaningful change. Workflow files live in [`.github/workflows`](https://github.com/apache/ozone/tree/master/.github/workflows) in [`apache/ozone`](https://github.com/apache/ozone). A longer, file-by-file reference lives in [`.github/ci.md`](https://github.com/apache/ozone/blob/master/.github/ci.md). -- [ci.md](https://github.com/apache/ozone/blob/master/.github/ci.md) -- [CONTRIBUTING.md](https://github.com/apache/ozone/blob/master/CONTRIBUTING.md#check-your-contribution) -- https://cwiki.apache.org/confluence/display/OZONE/Ozone+CI+with+Github+Actions -- https://cwiki.apache.org/confluence/display/OZONE/Github+Actions+tips+and+tricks. +:::info Use the right repository + +This page is about **[`apache/ozone`](https://github.com/apache/ozone)** (the Ozone product source code). The documentation site you are reading comes from **[`apache/ozone-site`](https://github.com/apache/ozone-site)** and has its **own** CI. For website-only edits, use the [ozone-site contributing guide](https://github.com/apache/ozone-site/blob/master/CONTRIBUTING.md). + +::: + +## Start here: your first code contribution + +Follow these steps once; after that, pushing to your branch is the usual loop. + +1. **Fork and clone** [`apache/ozone`](https://github.com/apache/ozone) to **your** GitHub account, then clone **your fork** locally. You will push branches to `origin` on the fork, then open a PR to `apache/ozone`. +2. **Turn on Actions** on the fork so workflows actually run ([how to enable them](#enable-github-actions-on-your-fork)). +3. **Jira** — Create or choose an issue in [HDDS](https://issues.apache.org/jira/projects/HDDS/) (the Ozone Jira project; the name is historical). Need an account? Use the ASF [Jira self-service](https://selfserve.apache.org/jira-account.html?project=ozone) form. +4. **Branch** — Work on a branch, often named after the issue (for example `HDDS-1234`). +5. **Push** — When you push, GitHub should show a **`build-branch`** workflow run under the **Actions** tab on your fork. Wait for it to finish and fix any failures you can reproduce. +6. **Open the PR** — Use the [pull request template](https://github.com/apache/ozone/blob/master/.github/pull_request_template.md). When the change is ready for review, set the Jira to **Patch Available** so committers know to look. + +The full narrative (reviews, merging, Jira etiquette) is in the [Ozone contributing guide](https://github.com/apache/ozone/blob/master/CONTRIBUTING.md#contribute-your-modifications). + +:::tip You can lean on CI first + +Many contributors fix quick issues by reading the failing log on GitHub, then pushing a small follow-up commit. Running every check locally is **optional but helpful** for faster feedback; see [Run checks on your machine](#run-checks-on-your-machine). + +::: + +## Enable GitHub Actions on your fork + +New forks sometimes have workflows off until you allow them. + +1. Open **your fork** on GitHub → **Settings** → **Actions** → **General**. +2. Under **Actions permissions**, pick a policy that allows workflows to run (many people use **Allow all actions and reusable workflows** on personal forks). +3. Open the **Actions** tab. If GitHub asks to enable workflows, confirm so **`build-branch`** runs when you push. + +More detail: [Enabling or disabling GitHub Actions](https://docs.github.com/en/repositories/managing-your-repositorys-settings-and-features/enabling-or-disabling-github-actions-for-a-repository). + +## What you see in GitHub: `build-branch` + +Two names show up in docs; both mean “the main CI pipeline”: + +| What | Meaning | +| --- | --- | +| **`build-branch`** | The **name** of the workflow in the Actions tab. It comes from the `name:` field in [`post-commit.yml`](https://github.com/apache/ozone/blob/master/.github/workflows/post-commit.yml). | +| **`ci.yml`** | Where most **jobs** (compile, tests, and so on) are defined. `post-commit.yml` calls this file as a [reusable workflow](https://docs.github.com/en/actions/using-workflows/reusing-workflows). | + +So: **`post-commit.yml`** = front door; **`ci.yml`** = where the heavy lifting is described. + +When you push new commits to an open pull request, **newer runs can cancel older ones** still in progress ([concurrency](https://docs.github.com/en/actions/using-jobs/using-concurrency)). That is normal and saves time. + +## Run checks on your machine + +Running scripts locally catches problems before you push. You need a working dev environment first—see [Build with Maven](../../developer-guide/build/maven) and [Building from source](https://github.com/apache/ozone/blob/master/CONTRIBUTING.md#building-from-source) in `CONTRIBUTING.md`. + +From the **root of your clone** (the folder that contains `hadoop-ozone/`): + +```bash +./hadoop-ozone/dev-support/checks/build.sh +``` + +Most checks live in [`hadoop-ozone/dev-support/checks`](https://github.com/apache/ozone/tree/master/hadoop-ozone/dev-support/checks). The [Check your contribution](https://github.com/apache/ozone/blob/master/CONTRIBUTING.md#check-your-contribution) section groups them by rough duration: + +| Rough time | Scripts | What they do | +| --- | --- | --- | +| Build step | `build.sh` | Compile Ozone | +| Minutes | `author.sh`, `bats.sh`, `rat.sh`, `docs.sh`, `dependency.sh`, `checkstyle.sh`, `pmd.sh` | Style, license headers, docs, dependency list | +| ~10 minutes | `findbugs.sh`, `kubernetes.sh` | SpotBugs, small Kubernetes-related checks | +| An hour or more | `unit.sh`, `integration.sh`, `acceptance.sh` | Unit tests, mini-cluster tests, Docker Compose acceptance tests | + +More on test styles: [Acceptance tests](./acceptance-tests) on this site. For unit and integration testing (and running checks locally), see [Running Ozone smoke tests and unit tests](https://cwiki.apache.org/confluence/display/OZONE/Running+Ozone+Smoke+Tests+and+Unit+Tests) on the wiki until the dedicated **Unit tests** and **Integration tests** pages here are published. + +`integration.sh` and `acceptance.sh` can take extra arguments to run a subset; open the scripts to see options. Output usually lands under `target/` (for example `target/docs`). + +## Why did CI skip some jobs? + +Not every pull request runs every job. A step called **build-info** runs [`selective_ci_checks.sh`](https://github.com/apache/ozone/blob/master/dev-support/ci/selective_ci_checks.sh) and only enables jobs that match the files you changed—unless: + +- the run is **not** from a PR, or +- the PR has the **`full tests needed`** [label](https://docs.github.com/en/issues/using-labels-and-milestones-to-track-work/managing-labels). + +So a focused change might show fewer checks than a large refactor. **That is expected.** Reviewers can add **`full tests needed`** when the full matrix is required. If you think the wrong jobs were skipped, **ask on the PR**; reviewers are used to that question. + +## What the main CI jobs do (overview) + +The list below matches [`.github/ci.md`](https://github.com/apache/ozone/blob/master/.github/ci.md). Treat it as a map when reading logs, not something to memorize. + +- **build-info** — Decides which other jobs run (selective CI). +- **compile** — Builds with Java 8 and 11 via [`build.sh`](https://github.com/apache/ozone/blob/master/hadoop-ozone/dev-support/checks/build.sh); later jobs typically use the Java 8 build. +- **basic** — Checks like author tags, BATS, Checkstyle, Hugo for docs, SpotBugs, PMD, RAT—depending on what was selected. +- **unit** — [`unit.sh`](https://github.com/apache/ozone/blob/master/hadoop-ozone/dev-support/checks/unit.sh) and [`native.sh`](https://github.com/apache/ozone/blob/master/hadoop-ozone/dev-support/checks/native.sh) for RocksDB-native tests. +- **dependency** — Compares JARs to [`jar-report.txt`](https://github.com/apache/ozone/blob/master/hadoop-ozone/dist/src/main/license/jar-report.txt). +- **acceptance** — [`acceptance.sh`](https://github.com/apache/ozone/blob/master/hadoop-ozone/dev-support/checks/acceptance.sh) (Robot Framework + Docker Compose; variants like secure / unsecure / misc). +- **kubernetes** — [`kubernetes.sh`](https://github.com/apache/ozone/blob/master/hadoop-ozone/dev-support/checks/kubernetes.sh). +- **integration** — [`integration.sh`](https://github.com/apache/ozone/blob/master/hadoop-ozone/dev-support/checks/integration.sh) (mini-cluster style tests, often sharded in CI). +- **coverage** — [`coverage.sh`](https://github.com/apache/ozone/blob/master/hadoop-ozone/dev-support/checks/coverage.sh) merges coverage when earlier jobs produced it. + +Some jobs use a **matrix** (for example multiple Java versions) with **fail-fast**: if one matrix leg fails, the others in that matrix stop. Unrelated jobs can still run until they finish or fail. + +## Other workflows + +The [workflows directory](https://github.com/apache/ozone/tree/master/.github/workflows) also contains jobs for caches, labeling, Ratis builds, repeating tests, generated docs, and more. The folder on `master` is the up-to-date list. + +### Stale pull requests + +[`close-stale-prs.yaml`](https://github.com/apache/ozone/blob/master/.github/workflows/close-stale-prs.yaml) runs on a timer and uses [actions/stale](https://github.com/actions/stale) to nudge and eventually close very inactive PRs. Exact timings are in that file. + +## If something fails + +:::note Green CI is a team norm + +A red check does not mean you did something wrong—it means the run found something to fix. It happens to everyone. + +::: + +1. Open the failed **`build-branch`** run → click the red job → read the **log** from the bottom upward for the first error. +2. If the job uploaded **Artifacts**, download them from the run summary (they expire after a short time). +3. Try the same **check script** locally if you have the environment set up ([Run checks on your machine](#run-checks-on-your-machine)). +4. To re-run without new code, use **Re-run all jobs** or **Re-run failed jobs** on the run page, or: + +```bash +git commit --allow-empty -m 'trigger new CI check' +``` + +Failures on the **main** repo’s default branch sometimes leave extra artifacts on [ozone build results](https://elek.github.io/ozone-build-results/) (community mirror). + +### Get help + +- Ask on your **pull request**—reviewers can interpret unfamiliar failures quickly. +- **Email** [[email protected]](mailto:[email protected]) for broader questions. +- **[GitHub Discussions](https://github.com/apache/ozone/discussions)** and the [#ozone](http://s.apache.org/slack-invite) Slack channel (ASF Slack) are listed in [`CONTRIBUTING.md`](https://github.com/apache/ozone/blob/master/CONTRIBUTING.md#who-to-contact). + +### Local and wiki testing notes + +- Wiki: [Running Ozone smoke tests and unit tests](https://cwiki.apache.org/confluence/display/OZONE/Running+Ozone+Smoke+Tests+and+Unit+Tests). + +## Advanced: flaky tests and debugging on a fork + +These patterns are for **repeat failures** or **environment-only** bugs. They usually live on a **personal fork**, not in `apache/ozone`. + +- Wiki: [GitHub Actions tips and tricks](https://cwiki.apache.org/confluence/display/OZONE/Github+Actions+tips+and+tricks) — running one test many times, extra logging, optional [tmate](https://github.com/tmate-io/tmate)-style access (unsafe on public repos; never with secrets exposed). +- Prefer current runner images (for example `ubuntu-latest`) when copying older examples. + +## Deprecated workflows + +Old workflows can still appear on the [Actions](https://github.com/apache/ozone/actions) tab. An outdated workflow also named **build-branch** is tied to [`chaos.yml`](https://github.com/apache/ozone/actions/workflows/chaos.yml), not the current `post-commit.yml` pipeline—compare URLs. Full list: [`.github/ci.md` — Old/Deprecated Workflows](https://github.com/apache/ozone/blob/master/.github/ci.md#olddeprecated-workflows). Review Comment: This can be removed, the confusing duplicate `build-branch` workflow no longer appears in https://github.com/apache/ozone/actions ########## docs/08-developer-guide/03-test/04-continuous-integration.md: ########## @@ -1,15 +1,160 @@ --- -draft: true sidebar_label: Continuous Integration --- # Continuous Integration With GitHub Actions -**TODO:** File a subtask under [HDDS-9861](https://issues.apache.org/jira/browse/HDDS-9861) and complete this page or section. +If you are new to the project, **you do not need to understand every job** below on day one. The goal of this page is to help you get a green **`build-branch`** run on your fork, know where to look when something fails, and find deeper detail when you need it. -Aggregate content from our various GitHub actions guides, including +Apache Ozone uses [GitHub Actions](https://docs.github.com/en/actions) to build and test every meaningful change. Workflow files live in [`.github/workflows`](https://github.com/apache/ozone/tree/master/.github/workflows) in [`apache/ozone`](https://github.com/apache/ozone). A longer, file-by-file reference lives in [`.github/ci.md`](https://github.com/apache/ozone/blob/master/.github/ci.md). -- [ci.md](https://github.com/apache/ozone/blob/master/.github/ci.md) -- [CONTRIBUTING.md](https://github.com/apache/ozone/blob/master/CONTRIBUTING.md#check-your-contribution) -- https://cwiki.apache.org/confluence/display/OZONE/Ozone+CI+with+Github+Actions -- https://cwiki.apache.org/confluence/display/OZONE/Github+Actions+tips+and+tricks. +:::info Use the right repository + +This page is about **[`apache/ozone`](https://github.com/apache/ozone)** (the Ozone product source code). The documentation site you are reading comes from **[`apache/ozone-site`](https://github.com/apache/ozone-site)** and has its **own** CI. For website-only edits, use the [ozone-site contributing guide](https://github.com/apache/ozone-site/blob/master/CONTRIBUTING.md). + +::: + +## Start here: your first code contribution + +Follow these steps once; after that, pushing to your branch is the usual loop. + +1. **Fork and clone** [`apache/ozone`](https://github.com/apache/ozone) to **your** GitHub account, then clone **your fork** locally. You will push branches to `origin` on the fork, then open a PR to `apache/ozone`. +2. **Turn on Actions** on the fork so workflows actually run ([how to enable them](#enable-github-actions-on-your-fork)). +3. **Jira** — Create or choose an issue in [HDDS](https://issues.apache.org/jira/projects/HDDS/) (the Ozone Jira project; the name is historical). Need an account? Use the ASF [Jira self-service](https://selfserve.apache.org/jira-account.html?project=ozone) form. +4. **Branch** — Work on a branch, often named after the issue (for example `HDDS-1234`). +5. **Push** — When you push, GitHub should show a **`build-branch`** workflow run under the **Actions** tab on your fork. Wait for it to finish and fix any failures you can reproduce. +6. **Open the PR** — Use the [pull request template](https://github.com/apache/ozone/blob/master/.github/pull_request_template.md). When the change is ready for review, set the Jira to **Patch Available** so committers know to look. + +The full narrative (reviews, merging, Jira etiquette) is in the [Ozone contributing guide](https://github.com/apache/ozone/blob/master/CONTRIBUTING.md#contribute-your-modifications). + +:::tip You can lean on CI first + +Many contributors fix quick issues by reading the failing log on GitHub, then pushing a small follow-up commit. Running every check locally is **optional but helpful** for faster feedback; see [Run checks on your machine](#run-checks-on-your-machine). + +::: + +## Enable GitHub Actions on your fork + +New forks sometimes have workflows off until you allow them. + +1. Open **your fork** on GitHub → **Settings** → **Actions** → **General**. +2. Under **Actions permissions**, pick a policy that allows workflows to run (many people use **Allow all actions and reusable workflows** on personal forks). +3. Open the **Actions** tab. If GitHub asks to enable workflows, confirm so **`build-branch`** runs when you push. + +More detail: [Enabling or disabling GitHub Actions](https://docs.github.com/en/repositories/managing-your-repositorys-settings-and-features/enabling-or-disabling-github-actions-for-a-repository). + +## What you see in GitHub: `build-branch` + +Two names show up in docs; both mean “the main CI pipeline”: + +| What | Meaning | +| --- | --- | +| **`build-branch`** | The **name** of the workflow in the Actions tab. It comes from the `name:` field in [`post-commit.yml`](https://github.com/apache/ozone/blob/master/.github/workflows/post-commit.yml). | +| **`ci.yml`** | Where most **jobs** (compile, tests, and so on) are defined. `post-commit.yml` calls this file as a [reusable workflow](https://docs.github.com/en/actions/using-workflows/reusing-workflows). | + +So: **`post-commit.yml`** = front door; **`ci.yml`** = where the heavy lifting is described. + +When you push new commits to an open pull request, **newer runs can cancel older ones** still in progress ([concurrency](https://docs.github.com/en/actions/using-jobs/using-concurrency)). That is normal and saves time. + +## Run checks on your machine + +Running scripts locally catches problems before you push. You need a working dev environment first—see [Build with Maven](../../developer-guide/build/maven) and [Building from source](https://github.com/apache/ozone/blob/master/CONTRIBUTING.md#building-from-source) in `CONTRIBUTING.md`. + +From the **root of your clone** (the folder that contains `hadoop-ozone/`): + +```bash +./hadoop-ozone/dev-support/checks/build.sh +``` + +Most checks live in [`hadoop-ozone/dev-support/checks`](https://github.com/apache/ozone/tree/master/hadoop-ozone/dev-support/checks). The [Check your contribution](https://github.com/apache/ozone/blob/master/CONTRIBUTING.md#check-your-contribution) section groups them by rough duration: + +| Rough time | Scripts | What they do | +| --- | --- | --- | +| Build step | `build.sh` | Compile Ozone | +| Minutes | `author.sh`, `bats.sh`, `rat.sh`, `docs.sh`, `dependency.sh`, `checkstyle.sh`, `pmd.sh` | Style, license headers, docs, dependency list | +| ~10 minutes | `findbugs.sh`, `kubernetes.sh` | SpotBugs, small Kubernetes-related checks | +| An hour or more | `unit.sh`, `integration.sh`, `acceptance.sh` | Unit tests, mini-cluster tests, Docker Compose acceptance tests | Review Comment: Omit `unit.sh` ########## docs/08-developer-guide/03-test/04-continuous-integration.md: ########## @@ -1,15 +1,160 @@ --- -draft: true sidebar_label: Continuous Integration --- # Continuous Integration With GitHub Actions -**TODO:** File a subtask under [HDDS-9861](https://issues.apache.org/jira/browse/HDDS-9861) and complete this page or section. +If you are new to the project, **you do not need to understand every job** below on day one. The goal of this page is to help you get a green **`build-branch`** run on your fork, know where to look when something fails, and find deeper detail when you need it. -Aggregate content from our various GitHub actions guides, including +Apache Ozone uses [GitHub Actions](https://docs.github.com/en/actions) to build and test every meaningful change. Workflow files live in [`.github/workflows`](https://github.com/apache/ozone/tree/master/.github/workflows) in [`apache/ozone`](https://github.com/apache/ozone). A longer, file-by-file reference lives in [`.github/ci.md`](https://github.com/apache/ozone/blob/master/.github/ci.md). -- [ci.md](https://github.com/apache/ozone/blob/master/.github/ci.md) -- [CONTRIBUTING.md](https://github.com/apache/ozone/blob/master/CONTRIBUTING.md#check-your-contribution) -- https://cwiki.apache.org/confluence/display/OZONE/Ozone+CI+with+Github+Actions -- https://cwiki.apache.org/confluence/display/OZONE/Github+Actions+tips+and+tricks. +:::info Use the right repository + +This page is about **[`apache/ozone`](https://github.com/apache/ozone)** (the Ozone product source code). The documentation site you are reading comes from **[`apache/ozone-site`](https://github.com/apache/ozone-site)** and has its **own** CI. For website-only edits, use the [ozone-site contributing guide](https://github.com/apache/ozone-site/blob/master/CONTRIBUTING.md). + +::: + +## Start here: your first code contribution + +Follow these steps once; after that, pushing to your branch is the usual loop. + +1. **Fork and clone** [`apache/ozone`](https://github.com/apache/ozone) to **your** GitHub account, then clone **your fork** locally. You will push branches to `origin` on the fork, then open a PR to `apache/ozone`. +2. **Turn on Actions** on the fork so workflows actually run ([how to enable them](#enable-github-actions-on-your-fork)). +3. **Jira** — Create or choose an issue in [HDDS](https://issues.apache.org/jira/projects/HDDS/) (the Ozone Jira project; the name is historical). Need an account? Use the ASF [Jira self-service](https://selfserve.apache.org/jira-account.html?project=ozone) form. +4. **Branch** — Work on a branch, often named after the issue (for example `HDDS-1234`). +5. **Push** — When you push, GitHub should show a **`build-branch`** workflow run under the **Actions** tab on your fork. Wait for it to finish and fix any failures you can reproduce. +6. **Open the PR** — Use the [pull request template](https://github.com/apache/ozone/blob/master/.github/pull_request_template.md). When the change is ready for review, set the Jira to **Patch Available** so committers know to look. + +The full narrative (reviews, merging, Jira etiquette) is in the [Ozone contributing guide](https://github.com/apache/ozone/blob/master/CONTRIBUTING.md#contribute-your-modifications). + +:::tip You can lean on CI first + +Many contributors fix quick issues by reading the failing log on GitHub, then pushing a small follow-up commit. Running every check locally is **optional but helpful** for faster feedback; see [Run checks on your machine](#run-checks-on-your-machine). + +::: + +## Enable GitHub Actions on your fork + +New forks sometimes have workflows off until you allow them. + +1. Open **your fork** on GitHub → **Settings** → **Actions** → **General**. +2. Under **Actions permissions**, pick a policy that allows workflows to run (many people use **Allow all actions and reusable workflows** on personal forks). +3. Open the **Actions** tab. If GitHub asks to enable workflows, confirm so **`build-branch`** runs when you push. + +More detail: [Enabling or disabling GitHub Actions](https://docs.github.com/en/repositories/managing-your-repositorys-settings-and-features/enabling-or-disabling-github-actions-for-a-repository). + +## What you see in GitHub: `build-branch` + +Two names show up in docs; both mean “the main CI pipeline”: + +| What | Meaning | +| --- | --- | +| **`build-branch`** | The **name** of the workflow in the Actions tab. It comes from the `name:` field in [`post-commit.yml`](https://github.com/apache/ozone/blob/master/.github/workflows/post-commit.yml). | +| **`ci.yml`** | Where most **jobs** (compile, tests, and so on) are defined. `post-commit.yml` calls this file as a [reusable workflow](https://docs.github.com/en/actions/using-workflows/reusing-workflows). | + +So: **`post-commit.yml`** = front door; **`ci.yml`** = where the heavy lifting is described. + +When you push new commits to an open pull request, **newer runs can cancel older ones** still in progress ([concurrency](https://docs.github.com/en/actions/using-jobs/using-concurrency)). That is normal and saves time. + +## Run checks on your machine + +Running scripts locally catches problems before you push. You need a working dev environment first—see [Build with Maven](../../developer-guide/build/maven) and [Building from source](https://github.com/apache/ozone/blob/master/CONTRIBUTING.md#building-from-source) in `CONTRIBUTING.md`. + +From the **root of your clone** (the folder that contains `hadoop-ozone/`): + +```bash +./hadoop-ozone/dev-support/checks/build.sh +``` + +Most checks live in [`hadoop-ozone/dev-support/checks`](https://github.com/apache/ozone/tree/master/hadoop-ozone/dev-support/checks). The [Check your contribution](https://github.com/apache/ozone/blob/master/CONTRIBUTING.md#check-your-contribution) section groups them by rough duration: + +| Rough time | Scripts | What they do | +| --- | --- | --- | +| Build step | `build.sh` | Compile Ozone | +| Minutes | `author.sh`, `bats.sh`, `rat.sh`, `docs.sh`, `dependency.sh`, `checkstyle.sh`, `pmd.sh` | Style, license headers, docs, dependency list | +| ~10 minutes | `findbugs.sh`, `kubernetes.sh` | SpotBugs, small Kubernetes-related checks | +| An hour or more | `unit.sh`, `integration.sh`, `acceptance.sh` | Unit tests, mini-cluster tests, Docker Compose acceptance tests | + +More on test styles: [Acceptance tests](./acceptance-tests) on this site. For unit and integration testing (and running checks locally), see [Running Ozone smoke tests and unit tests](https://cwiki.apache.org/confluence/display/OZONE/Running+Ozone+Smoke+Tests+and+Unit+Tests) on the wiki until the dedicated **Unit tests** and **Integration tests** pages here are published. + +`integration.sh` and `acceptance.sh` can take extra arguments to run a subset; open the scripts to see options. Output usually lands under `target/` (for example `target/docs`). + +## Why did CI skip some jobs? + +Not every pull request runs every job. A step called **build-info** runs [`selective_ci_checks.sh`](https://github.com/apache/ozone/blob/master/dev-support/ci/selective_ci_checks.sh) and only enables jobs that match the files you changed—unless: + +- the run is **not** from a PR, or +- the PR has the **`full tests needed`** [label](https://docs.github.com/en/issues/using-labels-and-milestones-to-track-work/managing-labels). + +So a focused change might show fewer checks than a large refactor. **That is expected.** Reviewers can add **`full tests needed`** when the full matrix is required. If you think the wrong jobs were skipped, **ask on the PR**; reviewers are used to that question. + +## What the main CI jobs do (overview) + +The list below matches [`.github/ci.md`](https://github.com/apache/ozone/blob/master/.github/ci.md). Treat it as a map when reading logs, not something to memorize. + +- **build-info** — Decides which other jobs run (selective CI). +- **compile** — Builds with Java 8 and 11 via [`build.sh`](https://github.com/apache/ozone/blob/master/hadoop-ozone/dev-support/checks/build.sh); later jobs typically use the Java 8 build. +- **basic** — Checks like author tags, BATS, Checkstyle, Hugo for docs, SpotBugs, PMD, RAT—depending on what was selected. +- **unit** — [`unit.sh`](https://github.com/apache/ozone/blob/master/hadoop-ozone/dev-support/checks/unit.sh) and [`native.sh`](https://github.com/apache/ozone/blob/master/hadoop-ozone/dev-support/checks/native.sh) for RocksDB-native tests. Review Comment: `unit` check no longer exists. `integration` check now runs all JUnit tests, regardless of which submodule they live in (HDDS-9242). The native lib is built/used in all jobs (HDDS-12734). ########## docs/08-developer-guide/03-test/04-continuous-integration.md: ########## @@ -1,15 +1,160 @@ --- -draft: true sidebar_label: Continuous Integration --- # Continuous Integration With GitHub Actions -**TODO:** File a subtask under [HDDS-9861](https://issues.apache.org/jira/browse/HDDS-9861) and complete this page or section. +If you are new to the project, **you do not need to understand every job** below on day one. The goal of this page is to help you get a green **`build-branch`** run on your fork, know where to look when something fails, and find deeper detail when you need it. -Aggregate content from our various GitHub actions guides, including +Apache Ozone uses [GitHub Actions](https://docs.github.com/en/actions) to build and test every meaningful change. Workflow files live in [`.github/workflows`](https://github.com/apache/ozone/tree/master/.github/workflows) in [`apache/ozone`](https://github.com/apache/ozone). A longer, file-by-file reference lives in [`.github/ci.md`](https://github.com/apache/ozone/blob/master/.github/ci.md). -- [ci.md](https://github.com/apache/ozone/blob/master/.github/ci.md) -- [CONTRIBUTING.md](https://github.com/apache/ozone/blob/master/CONTRIBUTING.md#check-your-contribution) -- https://cwiki.apache.org/confluence/display/OZONE/Ozone+CI+with+Github+Actions -- https://cwiki.apache.org/confluence/display/OZONE/Github+Actions+tips+and+tricks. +:::info Use the right repository + +This page is about **[`apache/ozone`](https://github.com/apache/ozone)** (the Ozone product source code). The documentation site you are reading comes from **[`apache/ozone-site`](https://github.com/apache/ozone-site)** and has its **own** CI. For website-only edits, use the [ozone-site contributing guide](https://github.com/apache/ozone-site/blob/master/CONTRIBUTING.md). + +::: + +## Start here: your first code contribution + +Follow these steps once; after that, pushing to your branch is the usual loop. + +1. **Fork and clone** [`apache/ozone`](https://github.com/apache/ozone) to **your** GitHub account, then clone **your fork** locally. You will push branches to `origin` on the fork, then open a PR to `apache/ozone`. +2. **Turn on Actions** on the fork so workflows actually run ([how to enable them](#enable-github-actions-on-your-fork)). +3. **Jira** — Create or choose an issue in [HDDS](https://issues.apache.org/jira/projects/HDDS/) (the Ozone Jira project; the name is historical). Need an account? Use the ASF [Jira self-service](https://selfserve.apache.org/jira-account.html?project=ozone) form. +4. **Branch** — Work on a branch, often named after the issue (for example `HDDS-1234`). +5. **Push** — When you push, GitHub should show a **`build-branch`** workflow run under the **Actions** tab on your fork. Wait for it to finish and fix any failures you can reproduce. +6. **Open the PR** — Use the [pull request template](https://github.com/apache/ozone/blob/master/.github/pull_request_template.md). When the change is ready for review, set the Jira to **Patch Available** so committers know to look. + +The full narrative (reviews, merging, Jira etiquette) is in the [Ozone contributing guide](https://github.com/apache/ozone/blob/master/CONTRIBUTING.md#contribute-your-modifications). + +:::tip You can lean on CI first + +Many contributors fix quick issues by reading the failing log on GitHub, then pushing a small follow-up commit. Running every check locally is **optional but helpful** for faster feedback; see [Run checks on your machine](#run-checks-on-your-machine). + +::: + +## Enable GitHub Actions on your fork + +New forks sometimes have workflows off until you allow them. + +1. Open **your fork** on GitHub → **Settings** → **Actions** → **General**. +2. Under **Actions permissions**, pick a policy that allows workflows to run (many people use **Allow all actions and reusable workflows** on personal forks). +3. Open the **Actions** tab. If GitHub asks to enable workflows, confirm so **`build-branch`** runs when you push. + +More detail: [Enabling or disabling GitHub Actions](https://docs.github.com/en/repositories/managing-your-repositorys-settings-and-features/enabling-or-disabling-github-actions-for-a-repository). + +## What you see in GitHub: `build-branch` + +Two names show up in docs; both mean “the main CI pipeline”: + +| What | Meaning | +| --- | --- | +| **`build-branch`** | The **name** of the workflow in the Actions tab. It comes from the `name:` field in [`post-commit.yml`](https://github.com/apache/ozone/blob/master/.github/workflows/post-commit.yml). | +| **`ci.yml`** | Where most **jobs** (compile, tests, and so on) are defined. `post-commit.yml` calls this file as a [reusable workflow](https://docs.github.com/en/actions/using-workflows/reusing-workflows). | + +So: **`post-commit.yml`** = front door; **`ci.yml`** = where the heavy lifting is described. + +When you push new commits to an open pull request, **newer runs can cancel older ones** still in progress ([concurrency](https://docs.github.com/en/actions/using-jobs/using-concurrency)). That is normal and saves time. + +## Run checks on your machine + +Running scripts locally catches problems before you push. You need a working dev environment first—see [Build with Maven](../../developer-guide/build/maven) and [Building from source](https://github.com/apache/ozone/blob/master/CONTRIBUTING.md#building-from-source) in `CONTRIBUTING.md`. + +From the **root of your clone** (the folder that contains `hadoop-ozone/`): + +```bash +./hadoop-ozone/dev-support/checks/build.sh +``` + +Most checks live in [`hadoop-ozone/dev-support/checks`](https://github.com/apache/ozone/tree/master/hadoop-ozone/dev-support/checks). The [Check your contribution](https://github.com/apache/ozone/blob/master/CONTRIBUTING.md#check-your-contribution) section groups them by rough duration: + +| Rough time | Scripts | What they do | +| --- | --- | --- | +| Build step | `build.sh` | Compile Ozone | +| Minutes | `author.sh`, `bats.sh`, `rat.sh`, `docs.sh`, `dependency.sh`, `checkstyle.sh`, `pmd.sh` | Style, license headers, docs, dependency list | +| ~10 minutes | `findbugs.sh`, `kubernetes.sh` | SpotBugs, small Kubernetes-related checks | +| An hour or more | `unit.sh`, `integration.sh`, `acceptance.sh` | Unit tests, mini-cluster tests, Docker Compose acceptance tests | + +More on test styles: [Acceptance tests](./acceptance-tests) on this site. For unit and integration testing (and running checks locally), see [Running Ozone smoke tests and unit tests](https://cwiki.apache.org/confluence/display/OZONE/Running+Ozone+Smoke+Tests+and+Unit+Tests) on the wiki until the dedicated **Unit tests** and **Integration tests** pages here are published. + +`integration.sh` and `acceptance.sh` can take extra arguments to run a subset; open the scripts to see options. Output usually lands under `target/` (for example `target/docs`). + +## Why did CI skip some jobs? + +Not every pull request runs every job. A step called **build-info** runs [`selective_ci_checks.sh`](https://github.com/apache/ozone/blob/master/dev-support/ci/selective_ci_checks.sh) and only enables jobs that match the files you changed—unless: + +- the run is **not** from a PR, or +- the PR has the **`full tests needed`** [label](https://docs.github.com/en/issues/using-labels-and-milestones-to-track-work/managing-labels). + +So a focused change might show fewer checks than a large refactor. **That is expected.** Reviewers can add **`full tests needed`** when the full matrix is required. If you think the wrong jobs were skipped, **ask on the PR**; reviewers are used to that question. + +## What the main CI jobs do (overview) + +The list below matches [`.github/ci.md`](https://github.com/apache/ozone/blob/master/.github/ci.md). Treat it as a map when reading logs, not something to memorize. + +- **build-info** — Decides which other jobs run (selective CI). +- **compile** — Builds with Java 8 and 11 via [`build.sh`](https://github.com/apache/ozone/blob/master/hadoop-ozone/dev-support/checks/build.sh); later jobs typically use the Java 8 build. +- **basic** — Checks like author tags, BATS, Checkstyle, Hugo for docs, SpotBugs, PMD, RAT—depending on what was selected. +- **unit** — [`unit.sh`](https://github.com/apache/ozone/blob/master/hadoop-ozone/dev-support/checks/unit.sh) and [`native.sh`](https://github.com/apache/ozone/blob/master/hadoop-ozone/dev-support/checks/native.sh) for RocksDB-native tests. +- **dependency** — Compares JARs to [`jar-report.txt`](https://github.com/apache/ozone/blob/master/hadoop-ozone/dist/src/main/license/jar-report.txt). +- **acceptance** — [`acceptance.sh`](https://github.com/apache/ozone/blob/master/hadoop-ozone/dev-support/checks/acceptance.sh) (Robot Framework + Docker Compose; variants like secure / unsecure / misc). +- **kubernetes** — [`kubernetes.sh`](https://github.com/apache/ozone/blob/master/hadoop-ozone/dev-support/checks/kubernetes.sh). +- **integration** — [`integration.sh`](https://github.com/apache/ozone/blob/master/hadoop-ozone/dev-support/checks/integration.sh) (mini-cluster style tests, often sharded in CI). +- **coverage** — [`coverage.sh`](https://github.com/apache/ozone/blob/master/hadoop-ozone/dev-support/checks/coverage.sh) merges coverage when earlier jobs produced it. + +Some jobs use a **matrix** (for example multiple Java versions) with **fail-fast**: if one matrix leg fails, the others in that matrix stop. Unrelated jobs can still run until they finish or fail. + +## Other workflows + +The [workflows directory](https://github.com/apache/ozone/tree/master/.github/workflows) also contains jobs for caches, labeling, Ratis builds, repeating tests, generated docs, and more. The folder on `master` is the up-to-date list. + +### Stale pull requests + +[`close-stale-prs.yaml`](https://github.com/apache/ozone/blob/master/.github/workflows/close-stale-prs.yaml) runs on a timer and uses [actions/stale](https://github.com/actions/stale) to nudge and eventually close very inactive PRs. Exact timings are in that file. + +## If something fails + +:::note Green CI is a team norm + +A red check does not mean you did something wrong—it means the run found something to fix. It happens to everyone. + +::: + +1. Open the failed **`build-branch`** run → click the red job → read the **log** from the bottom upward for the first error. +2. If the job uploaded **Artifacts**, download them from the run summary (they expire after a short time). +3. Try the same **check script** locally if you have the environment set up ([Run checks on your machine](#run-checks-on-your-machine)). Review Comment: I think we should differentiate here: 1. basic: can be safely run locally, they are quick and do not need prior build 2. dependency/license: quick, but need build 3. repro: needs build 4. integration check: pass `-Dtest='...'` to restrict to specific JUnit test(s), or run them using other means (e.g. IDE) 5. acceptance check: needs build, and better run only the failed `test*.sh` (the one mentioned in the log, e.g. `ERROR: Test execution of ozone/test-legacy-bucket.sh is FAILED`) 6. kubernetes: tested only on Linux ########## docs/08-developer-guide/03-test/04-continuous-integration.md: ########## @@ -1,15 +1,160 @@ --- -draft: true sidebar_label: Continuous Integration --- # Continuous Integration With GitHub Actions -**TODO:** File a subtask under [HDDS-9861](https://issues.apache.org/jira/browse/HDDS-9861) and complete this page or section. +If you are new to the project, **you do not need to understand every job** below on day one. The goal of this page is to help you get a green **`build-branch`** run on your fork, know where to look when something fails, and find deeper detail when you need it. -Aggregate content from our various GitHub actions guides, including +Apache Ozone uses [GitHub Actions](https://docs.github.com/en/actions) to build and test every meaningful change. Workflow files live in [`.github/workflows`](https://github.com/apache/ozone/tree/master/.github/workflows) in [`apache/ozone`](https://github.com/apache/ozone). A longer, file-by-file reference lives in [`.github/ci.md`](https://github.com/apache/ozone/blob/master/.github/ci.md). -- [ci.md](https://github.com/apache/ozone/blob/master/.github/ci.md) -- [CONTRIBUTING.md](https://github.com/apache/ozone/blob/master/CONTRIBUTING.md#check-your-contribution) -- https://cwiki.apache.org/confluence/display/OZONE/Ozone+CI+with+Github+Actions -- https://cwiki.apache.org/confluence/display/OZONE/Github+Actions+tips+and+tricks. +:::info Use the right repository + +This page is about **[`apache/ozone`](https://github.com/apache/ozone)** (the Ozone product source code). The documentation site you are reading comes from **[`apache/ozone-site`](https://github.com/apache/ozone-site)** and has its **own** CI. For website-only edits, use the [ozone-site contributing guide](https://github.com/apache/ozone-site/blob/master/CONTRIBUTING.md). + +::: + +## Start here: your first code contribution + +Follow these steps once; after that, pushing to your branch is the usual loop. + +1. **Fork and clone** [`apache/ozone`](https://github.com/apache/ozone) to **your** GitHub account, then clone **your fork** locally. You will push branches to `origin` on the fork, then open a PR to `apache/ozone`. +2. **Turn on Actions** on the fork so workflows actually run ([how to enable them](#enable-github-actions-on-your-fork)). +3. **Jira** — Create or choose an issue in [HDDS](https://issues.apache.org/jira/projects/HDDS/) (the Ozone Jira project; the name is historical). Need an account? Use the ASF [Jira self-service](https://selfserve.apache.org/jira-account.html?project=ozone) form. +4. **Branch** — Work on a branch, often named after the issue (for example `HDDS-1234`). +5. **Push** — When you push, GitHub should show a **`build-branch`** workflow run under the **Actions** tab on your fork. Wait for it to finish and fix any failures you can reproduce. +6. **Open the PR** — Use the [pull request template](https://github.com/apache/ozone/blob/master/.github/pull_request_template.md). When the change is ready for review, set the Jira to **Patch Available** so committers know to look. + +The full narrative (reviews, merging, Jira etiquette) is in the [Ozone contributing guide](https://github.com/apache/ozone/blob/master/CONTRIBUTING.md#contribute-your-modifications). + +:::tip You can lean on CI first + +Many contributors fix quick issues by reading the failing log on GitHub, then pushing a small follow-up commit. Running every check locally is **optional but helpful** for faster feedback; see [Run checks on your machine](#run-checks-on-your-machine). + +::: + +## Enable GitHub Actions on your fork + +New forks sometimes have workflows off until you allow them. + +1. Open **your fork** on GitHub → **Settings** → **Actions** → **General**. +2. Under **Actions permissions**, pick a policy that allows workflows to run (many people use **Allow all actions and reusable workflows** on personal forks). +3. Open the **Actions** tab. If GitHub asks to enable workflows, confirm so **`build-branch`** runs when you push. + +More detail: [Enabling or disabling GitHub Actions](https://docs.github.com/en/repositories/managing-your-repositorys-settings-and-features/enabling-or-disabling-github-actions-for-a-repository). + +## What you see in GitHub: `build-branch` + +Two names show up in docs; both mean “the main CI pipeline”: + +| What | Meaning | +| --- | --- | +| **`build-branch`** | The **name** of the workflow in the Actions tab. It comes from the `name:` field in [`post-commit.yml`](https://github.com/apache/ozone/blob/master/.github/workflows/post-commit.yml). | +| **`ci.yml`** | Where most **jobs** (compile, tests, and so on) are defined. `post-commit.yml` calls this file as a [reusable workflow](https://docs.github.com/en/actions/using-workflows/reusing-workflows). | + +So: **`post-commit.yml`** = front door; **`ci.yml`** = where the heavy lifting is described. + +When you push new commits to an open pull request, **newer runs can cancel older ones** still in progress ([concurrency](https://docs.github.com/en/actions/using-jobs/using-concurrency)). That is normal and saves time. + +## Run checks on your machine + +Running scripts locally catches problems before you push. You need a working dev environment first—see [Build with Maven](../../developer-guide/build/maven) and [Building from source](https://github.com/apache/ozone/blob/master/CONTRIBUTING.md#building-from-source) in `CONTRIBUTING.md`. + +From the **root of your clone** (the folder that contains `hadoop-ozone/`): + +```bash +./hadoop-ozone/dev-support/checks/build.sh +``` + +Most checks live in [`hadoop-ozone/dev-support/checks`](https://github.com/apache/ozone/tree/master/hadoop-ozone/dev-support/checks). The [Check your contribution](https://github.com/apache/ozone/blob/master/CONTRIBUTING.md#check-your-contribution) section groups them by rough duration: + +| Rough time | Scripts | What they do | +| --- | --- | --- | +| Build step | `build.sh` | Compile Ozone | +| Minutes | `author.sh`, `bats.sh`, `rat.sh`, `docs.sh`, `dependency.sh`, `checkstyle.sh`, `pmd.sh` | Style, license headers, docs, dependency list | +| ~10 minutes | `findbugs.sh`, `kubernetes.sh` | SpotBugs, small Kubernetes-related checks | +| An hour or more | `unit.sh`, `integration.sh`, `acceptance.sh` | Unit tests, mini-cluster tests, Docker Compose acceptance tests | + +More on test styles: [Acceptance tests](./acceptance-tests) on this site. For unit and integration testing (and running checks locally), see [Running Ozone smoke tests and unit tests](https://cwiki.apache.org/confluence/display/OZONE/Running+Ozone+Smoke+Tests+and+Unit+Tests) on the wiki until the dedicated **Unit tests** and **Integration tests** pages here are published. + +`integration.sh` and `acceptance.sh` can take extra arguments to run a subset; open the scripts to see options. Output usually lands under `target/` (for example `target/docs`). + +## Why did CI skip some jobs? + +Not every pull request runs every job. A step called **build-info** runs [`selective_ci_checks.sh`](https://github.com/apache/ozone/blob/master/dev-support/ci/selective_ci_checks.sh) and only enables jobs that match the files you changed—unless: + +- the run is **not** from a PR, or +- the PR has the **`full tests needed`** [label](https://docs.github.com/en/issues/using-labels-and-milestones-to-track-work/managing-labels). + +So a focused change might show fewer checks than a large refactor. **That is expected.** Reviewers can add **`full tests needed`** when the full matrix is required. If you think the wrong jobs were skipped, **ask on the PR**; reviewers are used to that question. + +## What the main CI jobs do (overview) + +The list below matches [`.github/ci.md`](https://github.com/apache/ozone/blob/master/.github/ci.md). Treat it as a map when reading logs, not something to memorize. + +- **build-info** — Decides which other jobs run (selective CI). +- **compile** — Builds with Java 8 and 11 via [`build.sh`](https://github.com/apache/ozone/blob/master/hadoop-ozone/dev-support/checks/build.sh); later jobs typically use the Java 8 build. +- **basic** — Checks like author tags, BATS, Checkstyle, Hugo for docs, SpotBugs, PMD, RAT—depending on what was selected. +- **unit** — [`unit.sh`](https://github.com/apache/ozone/blob/master/hadoop-ozone/dev-support/checks/unit.sh) and [`native.sh`](https://github.com/apache/ozone/blob/master/hadoop-ozone/dev-support/checks/native.sh) for RocksDB-native tests. +- **dependency** — Compares JARs to [`jar-report.txt`](https://github.com/apache/ozone/blob/master/hadoop-ozone/dist/src/main/license/jar-report.txt). Review Comment: `dependency` checks if any dependencies are added or removed, as a reminder to update `LICENSE.txt`. (Using `jar-report.txt` is more of an implementation detail.) ########## docs/08-developer-guide/03-test/04-continuous-integration.md: ########## @@ -1,15 +1,160 @@ --- -draft: true sidebar_label: Continuous Integration --- # Continuous Integration With GitHub Actions -**TODO:** File a subtask under [HDDS-9861](https://issues.apache.org/jira/browse/HDDS-9861) and complete this page or section. +If you are new to the project, **you do not need to understand every job** below on day one. The goal of this page is to help you get a green **`build-branch`** run on your fork, know where to look when something fails, and find deeper detail when you need it. -Aggregate content from our various GitHub actions guides, including +Apache Ozone uses [GitHub Actions](https://docs.github.com/en/actions) to build and test every meaningful change. Workflow files live in [`.github/workflows`](https://github.com/apache/ozone/tree/master/.github/workflows) in [`apache/ozone`](https://github.com/apache/ozone). A longer, file-by-file reference lives in [`.github/ci.md`](https://github.com/apache/ozone/blob/master/.github/ci.md). -- [ci.md](https://github.com/apache/ozone/blob/master/.github/ci.md) -- [CONTRIBUTING.md](https://github.com/apache/ozone/blob/master/CONTRIBUTING.md#check-your-contribution) -- https://cwiki.apache.org/confluence/display/OZONE/Ozone+CI+with+Github+Actions -- https://cwiki.apache.org/confluence/display/OZONE/Github+Actions+tips+and+tricks. +:::info Use the right repository + +This page is about **[`apache/ozone`](https://github.com/apache/ozone)** (the Ozone product source code). The documentation site you are reading comes from **[`apache/ozone-site`](https://github.com/apache/ozone-site)** and has its **own** CI. For website-only edits, use the [ozone-site contributing guide](https://github.com/apache/ozone-site/blob/master/CONTRIBUTING.md). + +::: + +## Start here: your first code contribution + +Follow these steps once; after that, pushing to your branch is the usual loop. + +1. **Fork and clone** [`apache/ozone`](https://github.com/apache/ozone) to **your** GitHub account, then clone **your fork** locally. You will push branches to `origin` on the fork, then open a PR to `apache/ozone`. +2. **Turn on Actions** on the fork so workflows actually run ([how to enable them](#enable-github-actions-on-your-fork)). +3. **Jira** — Create or choose an issue in [HDDS](https://issues.apache.org/jira/projects/HDDS/) (the Ozone Jira project; the name is historical). Need an account? Use the ASF [Jira self-service](https://selfserve.apache.org/jira-account.html?project=ozone) form. +4. **Branch** — Work on a branch, often named after the issue (for example `HDDS-1234`). +5. **Push** — When you push, GitHub should show a **`build-branch`** workflow run under the **Actions** tab on your fork. Wait for it to finish and fix any failures you can reproduce. +6. **Open the PR** — Use the [pull request template](https://github.com/apache/ozone/blob/master/.github/pull_request_template.md). When the change is ready for review, set the Jira to **Patch Available** so committers know to look. + +The full narrative (reviews, merging, Jira etiquette) is in the [Ozone contributing guide](https://github.com/apache/ozone/blob/master/CONTRIBUTING.md#contribute-your-modifications). + +:::tip You can lean on CI first + +Many contributors fix quick issues by reading the failing log on GitHub, then pushing a small follow-up commit. Running every check locally is **optional but helpful** for faster feedback; see [Run checks on your machine](#run-checks-on-your-machine). + +::: + +## Enable GitHub Actions on your fork + +New forks sometimes have workflows off until you allow them. + +1. Open **your fork** on GitHub → **Settings** → **Actions** → **General**. +2. Under **Actions permissions**, pick a policy that allows workflows to run (many people use **Allow all actions and reusable workflows** on personal forks). +3. Open the **Actions** tab. If GitHub asks to enable workflows, confirm so **`build-branch`** runs when you push. + +More detail: [Enabling or disabling GitHub Actions](https://docs.github.com/en/repositories/managing-your-repositorys-settings-and-features/enabling-or-disabling-github-actions-for-a-repository). + +## What you see in GitHub: `build-branch` + +Two names show up in docs; both mean “the main CI pipeline”: + +| What | Meaning | +| --- | --- | +| **`build-branch`** | The **name** of the workflow in the Actions tab. It comes from the `name:` field in [`post-commit.yml`](https://github.com/apache/ozone/blob/master/.github/workflows/post-commit.yml). | +| **`ci.yml`** | Where most **jobs** (compile, tests, and so on) are defined. `post-commit.yml` calls this file as a [reusable workflow](https://docs.github.com/en/actions/using-workflows/reusing-workflows). | + +So: **`post-commit.yml`** = front door; **`ci.yml`** = where the heavy lifting is described. + +When you push new commits to an open pull request, **newer runs can cancel older ones** still in progress ([concurrency](https://docs.github.com/en/actions/using-jobs/using-concurrency)). That is normal and saves time. + +## Run checks on your machine + +Running scripts locally catches problems before you push. You need a working dev environment first—see [Build with Maven](../../developer-guide/build/maven) and [Building from source](https://github.com/apache/ozone/blob/master/CONTRIBUTING.md#building-from-source) in `CONTRIBUTING.md`. + +From the **root of your clone** (the folder that contains `hadoop-ozone/`): + +```bash +./hadoop-ozone/dev-support/checks/build.sh +``` + +Most checks live in [`hadoop-ozone/dev-support/checks`](https://github.com/apache/ozone/tree/master/hadoop-ozone/dev-support/checks). The [Check your contribution](https://github.com/apache/ozone/blob/master/CONTRIBUTING.md#check-your-contribution) section groups them by rough duration: + +| Rough time | Scripts | What they do | +| --- | --- | --- | +| Build step | `build.sh` | Compile Ozone | +| Minutes | `author.sh`, `bats.sh`, `rat.sh`, `docs.sh`, `dependency.sh`, `checkstyle.sh`, `pmd.sh` | Style, license headers, docs, dependency list | +| ~10 minutes | `findbugs.sh`, `kubernetes.sh` | SpotBugs, small Kubernetes-related checks | +| An hour or more | `unit.sh`, `integration.sh`, `acceptance.sh` | Unit tests, mini-cluster tests, Docker Compose acceptance tests | + +More on test styles: [Acceptance tests](./acceptance-tests) on this site. For unit and integration testing (and running checks locally), see [Running Ozone smoke tests and unit tests](https://cwiki.apache.org/confluence/display/OZONE/Running+Ozone+Smoke+Tests+and+Unit+Tests) on the wiki until the dedicated **Unit tests** and **Integration tests** pages here are published. + +`integration.sh` and `acceptance.sh` can take extra arguments to run a subset; open the scripts to see options. Output usually lands under `target/` (for example `target/docs`). + +## Why did CI skip some jobs? + +Not every pull request runs every job. A step called **build-info** runs [`selective_ci_checks.sh`](https://github.com/apache/ozone/blob/master/dev-support/ci/selective_ci_checks.sh) and only enables jobs that match the files you changed—unless: + +- the run is **not** from a PR, or +- the PR has the **`full tests needed`** [label](https://docs.github.com/en/issues/using-labels-and-milestones-to-track-work/managing-labels). + +So a focused change might show fewer checks than a large refactor. **That is expected.** Reviewers can add **`full tests needed`** when the full matrix is required. If you think the wrong jobs were skipped, **ask on the PR**; reviewers are used to that question. + +## What the main CI jobs do (overview) + +The list below matches [`.github/ci.md`](https://github.com/apache/ozone/blob/master/.github/ci.md). Treat it as a map when reading logs, not something to memorize. + +- **build-info** — Decides which other jobs run (selective CI). +- **compile** — Builds with Java 8 and 11 via [`build.sh`](https://github.com/apache/ozone/blob/master/hadoop-ozone/dev-support/checks/build.sh); later jobs typically use the Java 8 build. +- **basic** — Checks like author tags, BATS, Checkstyle, Hugo for docs, SpotBugs, PMD, RAT—depending on what was selected. +- **unit** — [`unit.sh`](https://github.com/apache/ozone/blob/master/hadoop-ozone/dev-support/checks/unit.sh) and [`native.sh`](https://github.com/apache/ozone/blob/master/hadoop-ozone/dev-support/checks/native.sh) for RocksDB-native tests. +- **dependency** — Compares JARs to [`jar-report.txt`](https://github.com/apache/ozone/blob/master/hadoop-ozone/dist/src/main/license/jar-report.txt). +- **acceptance** — [`acceptance.sh`](https://github.com/apache/ozone/blob/master/hadoop-ozone/dev-support/checks/acceptance.sh) (Robot Framework + Docker Compose; variants like secure / unsecure / misc). +- **kubernetes** — [`kubernetes.sh`](https://github.com/apache/ozone/blob/master/hadoop-ozone/dev-support/checks/kubernetes.sh). +- **integration** — [`integration.sh`](https://github.com/apache/ozone/blob/master/hadoop-ozone/dev-support/checks/integration.sh) (mini-cluster style tests, often sharded in CI). +- **coverage** — [`coverage.sh`](https://github.com/apache/ozone/blob/master/hadoop-ozone/dev-support/checks/coverage.sh) merges coverage when earlier jobs produced it. + +Some jobs use a **matrix** (for example multiple Java versions) with **fail-fast**: if one matrix leg fails, the others in that matrix stop. Unrelated jobs can still run until they finish or fail. + +## Other workflows + +The [workflows directory](https://github.com/apache/ozone/tree/master/.github/workflows) also contains jobs for caches, labeling, Ratis builds, repeating tests, generated docs, and more. The folder on `master` is the up-to-date list. + +### Stale pull requests + +[`close-stale-prs.yaml`](https://github.com/apache/ozone/blob/master/.github/workflows/close-stale-prs.yaml) runs on a timer and uses [actions/stale](https://github.com/actions/stale) to nudge and eventually close very inactive PRs. Exact timings are in that file. + +## If something fails + +:::note Green CI is a team norm + +A red check does not mean you did something wrong—it means the run found something to fix. It happens to everyone. + +::: + +1. Open the failed **`build-branch`** run → click the red job → read the **log** from the bottom upward for the first error. +2. If the job uploaded **Artifacts**, download them from the run summary (they expire after a short time). +3. Try the same **check script** locally if you have the environment set up ([Run checks on your machine](#run-checks-on-your-machine)). +4. To re-run without new code, use **Re-run all jobs** or **Re-run failed jobs** on the run page, or: + +```bash +git commit --allow-empty -m 'trigger new CI check' +``` + +Failures on the **main** repo’s default branch sometimes leave extra artifacts on [ozone build results](https://elek.github.io/ozone-build-results/) (community mirror). + +### Get help + +- Ask on your **pull request**—reviewers can interpret unfamiliar failures quickly. +- **Email** [[email protected]](mailto:[email protected]) for broader questions. +- **[GitHub Discussions](https://github.com/apache/ozone/discussions)** and the [#ozone](http://s.apache.org/slack-invite) Slack channel (ASF Slack) are listed in [`CONTRIBUTING.md`](https://github.com/apache/ozone/blob/master/CONTRIBUTING.md#who-to-contact). + +### Local and wiki testing notes + +- Wiki: [Running Ozone smoke tests and unit tests](https://cwiki.apache.org/confluence/display/OZONE/Running+Ozone+Smoke+Tests+and+Unit+Tests). Review Comment: I think we can omit this, content is outdated. ########## docs/08-developer-guide/03-test/04-continuous-integration.md: ########## @@ -1,15 +1,160 @@ --- -draft: true sidebar_label: Continuous Integration --- # Continuous Integration With GitHub Actions -**TODO:** File a subtask under [HDDS-9861](https://issues.apache.org/jira/browse/HDDS-9861) and complete this page or section. +If you are new to the project, **you do not need to understand every job** below on day one. The goal of this page is to help you get a green **`build-branch`** run on your fork, know where to look when something fails, and find deeper detail when you need it. -Aggregate content from our various GitHub actions guides, including +Apache Ozone uses [GitHub Actions](https://docs.github.com/en/actions) to build and test every meaningful change. Workflow files live in [`.github/workflows`](https://github.com/apache/ozone/tree/master/.github/workflows) in [`apache/ozone`](https://github.com/apache/ozone). A longer, file-by-file reference lives in [`.github/ci.md`](https://github.com/apache/ozone/blob/master/.github/ci.md). -- [ci.md](https://github.com/apache/ozone/blob/master/.github/ci.md) -- [CONTRIBUTING.md](https://github.com/apache/ozone/blob/master/CONTRIBUTING.md#check-your-contribution) -- https://cwiki.apache.org/confluence/display/OZONE/Ozone+CI+with+Github+Actions -- https://cwiki.apache.org/confluence/display/OZONE/Github+Actions+tips+and+tricks. +:::info Use the right repository + +This page is about **[`apache/ozone`](https://github.com/apache/ozone)** (the Ozone product source code). The documentation site you are reading comes from **[`apache/ozone-site`](https://github.com/apache/ozone-site)** and has its **own** CI. For website-only edits, use the [ozone-site contributing guide](https://github.com/apache/ozone-site/blob/master/CONTRIBUTING.md). + +::: + +## Start here: your first code contribution + +Follow these steps once; after that, pushing to your branch is the usual loop. + +1. **Fork and clone** [`apache/ozone`](https://github.com/apache/ozone) to **your** GitHub account, then clone **your fork** locally. You will push branches to `origin` on the fork, then open a PR to `apache/ozone`. +2. **Turn on Actions** on the fork so workflows actually run ([how to enable them](#enable-github-actions-on-your-fork)). +3. **Jira** — Create or choose an issue in [HDDS](https://issues.apache.org/jira/projects/HDDS/) (the Ozone Jira project; the name is historical). Need an account? Use the ASF [Jira self-service](https://selfserve.apache.org/jira-account.html?project=ozone) form. +4. **Branch** — Work on a branch, often named after the issue (for example `HDDS-1234`). +5. **Push** — When you push, GitHub should show a **`build-branch`** workflow run under the **Actions** tab on your fork. Wait for it to finish and fix any failures you can reproduce. +6. **Open the PR** — Use the [pull request template](https://github.com/apache/ozone/blob/master/.github/pull_request_template.md). When the change is ready for review, set the Jira to **Patch Available** so committers know to look. + +The full narrative (reviews, merging, Jira etiquette) is in the [Ozone contributing guide](https://github.com/apache/ozone/blob/master/CONTRIBUTING.md#contribute-your-modifications). + +:::tip You can lean on CI first + +Many contributors fix quick issues by reading the failing log on GitHub, then pushing a small follow-up commit. Running every check locally is **optional but helpful** for faster feedback; see [Run checks on your machine](#run-checks-on-your-machine). + +::: + +## Enable GitHub Actions on your fork + +New forks sometimes have workflows off until you allow them. + +1. Open **your fork** on GitHub → **Settings** → **Actions** → **General**. +2. Under **Actions permissions**, pick a policy that allows workflows to run (many people use **Allow all actions and reusable workflows** on personal forks). +3. Open the **Actions** tab. If GitHub asks to enable workflows, confirm so **`build-branch`** runs when you push. + +More detail: [Enabling or disabling GitHub Actions](https://docs.github.com/en/repositories/managing-your-repositorys-settings-and-features/enabling-or-disabling-github-actions-for-a-repository). + +## What you see in GitHub: `build-branch` + +Two names show up in docs; both mean “the main CI pipeline”: + +| What | Meaning | +| --- | --- | +| **`build-branch`** | The **name** of the workflow in the Actions tab. It comes from the `name:` field in [`post-commit.yml`](https://github.com/apache/ozone/blob/master/.github/workflows/post-commit.yml). | +| **`ci.yml`** | Where most **jobs** (compile, tests, and so on) are defined. `post-commit.yml` calls this file as a [reusable workflow](https://docs.github.com/en/actions/using-workflows/reusing-workflows). | + +So: **`post-commit.yml`** = front door; **`ci.yml`** = where the heavy lifting is described. + +When you push new commits to an open pull request, **newer runs can cancel older ones** still in progress ([concurrency](https://docs.github.com/en/actions/using-jobs/using-concurrency)). That is normal and saves time. + +## Run checks on your machine + +Running scripts locally catches problems before you push. You need a working dev environment first—see [Build with Maven](../../developer-guide/build/maven) and [Building from source](https://github.com/apache/ozone/blob/master/CONTRIBUTING.md#building-from-source) in `CONTRIBUTING.md`. + +From the **root of your clone** (the folder that contains `hadoop-ozone/`): + +```bash +./hadoop-ozone/dev-support/checks/build.sh +``` Review Comment: We should highlight that this command is just an example. I'd also move it after the table below. ########## docs/08-developer-guide/03-test/04-continuous-integration.md: ########## @@ -1,15 +1,160 @@ --- -draft: true sidebar_label: Continuous Integration --- # Continuous Integration With GitHub Actions -**TODO:** File a subtask under [HDDS-9861](https://issues.apache.org/jira/browse/HDDS-9861) and complete this page or section. +If you are new to the project, **you do not need to understand every job** below on day one. The goal of this page is to help you get a green **`build-branch`** run on your fork, know where to look when something fails, and find deeper detail when you need it. -Aggregate content from our various GitHub actions guides, including +Apache Ozone uses [GitHub Actions](https://docs.github.com/en/actions) to build and test every meaningful change. Workflow files live in [`.github/workflows`](https://github.com/apache/ozone/tree/master/.github/workflows) in [`apache/ozone`](https://github.com/apache/ozone). A longer, file-by-file reference lives in [`.github/ci.md`](https://github.com/apache/ozone/blob/master/.github/ci.md). -- [ci.md](https://github.com/apache/ozone/blob/master/.github/ci.md) -- [CONTRIBUTING.md](https://github.com/apache/ozone/blob/master/CONTRIBUTING.md#check-your-contribution) -- https://cwiki.apache.org/confluence/display/OZONE/Ozone+CI+with+Github+Actions -- https://cwiki.apache.org/confluence/display/OZONE/Github+Actions+tips+and+tricks. +:::info Use the right repository + +This page is about **[`apache/ozone`](https://github.com/apache/ozone)** (the Ozone product source code). The documentation site you are reading comes from **[`apache/ozone-site`](https://github.com/apache/ozone-site)** and has its **own** CI. For website-only edits, use the [ozone-site contributing guide](https://github.com/apache/ozone-site/blob/master/CONTRIBUTING.md). + +::: + +## Start here: your first code contribution + +Follow these steps once; after that, pushing to your branch is the usual loop. + +1. **Fork and clone** [`apache/ozone`](https://github.com/apache/ozone) to **your** GitHub account, then clone **your fork** locally. You will push branches to `origin` on the fork, then open a PR to `apache/ozone`. +2. **Turn on Actions** on the fork so workflows actually run ([how to enable them](#enable-github-actions-on-your-fork)). +3. **Jira** — Create or choose an issue in [HDDS](https://issues.apache.org/jira/projects/HDDS/) (the Ozone Jira project; the name is historical). Need an account? Use the ASF [Jira self-service](https://selfserve.apache.org/jira-account.html?project=ozone) form. +4. **Branch** — Work on a branch, often named after the issue (for example `HDDS-1234`). +5. **Push** — When you push, GitHub should show a **`build-branch`** workflow run under the **Actions** tab on your fork. Wait for it to finish and fix any failures you can reproduce. +6. **Open the PR** — Use the [pull request template](https://github.com/apache/ozone/blob/master/.github/pull_request_template.md). When the change is ready for review, set the Jira to **Patch Available** so committers know to look. + +The full narrative (reviews, merging, Jira etiquette) is in the [Ozone contributing guide](https://github.com/apache/ozone/blob/master/CONTRIBUTING.md#contribute-your-modifications). + +:::tip You can lean on CI first + +Many contributors fix quick issues by reading the failing log on GitHub, then pushing a small follow-up commit. Running every check locally is **optional but helpful** for faster feedback; see [Run checks on your machine](#run-checks-on-your-machine). + +::: + +## Enable GitHub Actions on your fork + +New forks sometimes have workflows off until you allow them. + +1. Open **your fork** on GitHub → **Settings** → **Actions** → **General**. +2. Under **Actions permissions**, pick a policy that allows workflows to run (many people use **Allow all actions and reusable workflows** on personal forks). +3. Open the **Actions** tab. If GitHub asks to enable workflows, confirm so **`build-branch`** runs when you push. + +More detail: [Enabling or disabling GitHub Actions](https://docs.github.com/en/repositories/managing-your-repositorys-settings-and-features/enabling-or-disabling-github-actions-for-a-repository). + +## What you see in GitHub: `build-branch` + +Two names show up in docs; both mean “the main CI pipeline”: + +| What | Meaning | +| --- | --- | +| **`build-branch`** | The **name** of the workflow in the Actions tab. It comes from the `name:` field in [`post-commit.yml`](https://github.com/apache/ozone/blob/master/.github/workflows/post-commit.yml). | +| **`ci.yml`** | Where most **jobs** (compile, tests, and so on) are defined. `post-commit.yml` calls this file as a [reusable workflow](https://docs.github.com/en/actions/using-workflows/reusing-workflows). | + +So: **`post-commit.yml`** = front door; **`ci.yml`** = where the heavy lifting is described. + +When you push new commits to an open pull request, **newer runs can cancel older ones** still in progress ([concurrency](https://docs.github.com/en/actions/using-jobs/using-concurrency)). That is normal and saves time. + +## Run checks on your machine + +Running scripts locally catches problems before you push. You need a working dev environment first—see [Build with Maven](../../developer-guide/build/maven) and [Building from source](https://github.com/apache/ozone/blob/master/CONTRIBUTING.md#building-from-source) in `CONTRIBUTING.md`. + +From the **root of your clone** (the folder that contains `hadoop-ozone/`): + +```bash +./hadoop-ozone/dev-support/checks/build.sh +``` + +Most checks live in [`hadoop-ozone/dev-support/checks`](https://github.com/apache/ozone/tree/master/hadoop-ozone/dev-support/checks). The [Check your contribution](https://github.com/apache/ozone/blob/master/CONTRIBUTING.md#check-your-contribution) section groups them by rough duration: + +| Rough time | Scripts | What they do | +| --- | --- | --- | +| Build step | `build.sh` | Compile Ozone | +| Minutes | `author.sh`, `bats.sh`, `rat.sh`, `docs.sh`, `dependency.sh`, `checkstyle.sh`, `pmd.sh` | Style, license headers, docs, dependency list | +| ~10 minutes | `findbugs.sh`, `kubernetes.sh` | SpotBugs, small Kubernetes-related checks | +| An hour or more | `unit.sh`, `integration.sh`, `acceptance.sh` | Unit tests, mini-cluster tests, Docker Compose acceptance tests | + +More on test styles: [Acceptance tests](./acceptance-tests) on this site. For unit and integration testing (and running checks locally), see [Running Ozone smoke tests and unit tests](https://cwiki.apache.org/confluence/display/OZONE/Running+Ozone+Smoke+Tests+and+Unit+Tests) on the wiki until the dedicated **Unit tests** and **Integration tests** pages here are published. + +`integration.sh` and `acceptance.sh` can take extra arguments to run a subset; open the scripts to see options. Output usually lands under `target/` (for example `target/docs`). + +## Why did CI skip some jobs? + +Not every pull request runs every job. A step called **build-info** runs [`selective_ci_checks.sh`](https://github.com/apache/ozone/blob/master/dev-support/ci/selective_ci_checks.sh) and only enables jobs that match the files you changed—unless: + +- the run is **not** from a PR, or +- the PR has the **`full tests needed`** [label](https://docs.github.com/en/issues/using-labels-and-milestones-to-track-work/managing-labels). + +So a focused change might show fewer checks than a large refactor. **That is expected.** Reviewers can add **`full tests needed`** when the full matrix is required. If you think the wrong jobs were skipped, **ask on the PR**; reviewers are used to that question. + +## What the main CI jobs do (overview) + +The list below matches [`.github/ci.md`](https://github.com/apache/ozone/blob/master/.github/ci.md). Treat it as a map when reading logs, not something to memorize. + +- **build-info** — Decides which other jobs run (selective CI). +- **compile** — Builds with Java 8 and 11 via [`build.sh`](https://github.com/apache/ozone/blob/master/hadoop-ozone/dev-support/checks/build.sh); later jobs typically use the Java 8 build. +- **basic** — Checks like author tags, BATS, Checkstyle, Hugo for docs, SpotBugs, PMD, RAT—depending on what was selected. +- **unit** — [`unit.sh`](https://github.com/apache/ozone/blob/master/hadoop-ozone/dev-support/checks/unit.sh) and [`native.sh`](https://github.com/apache/ozone/blob/master/hadoop-ozone/dev-support/checks/native.sh) for RocksDB-native tests. +- **dependency** — Compares JARs to [`jar-report.txt`](https://github.com/apache/ozone/blob/master/hadoop-ozone/dist/src/main/license/jar-report.txt). +- **acceptance** — [`acceptance.sh`](https://github.com/apache/ozone/blob/master/hadoop-ozone/dev-support/checks/acceptance.sh) (Robot Framework + Docker Compose; variants like secure / unsecure / misc). +- **kubernetes** — [`kubernetes.sh`](https://github.com/apache/ozone/blob/master/hadoop-ozone/dev-support/checks/kubernetes.sh). +- **integration** — [`integration.sh`](https://github.com/apache/ozone/blob/master/hadoop-ozone/dev-support/checks/integration.sh) (mini-cluster style tests, often sharded in CI). +- **coverage** — [`coverage.sh`](https://github.com/apache/ozone/blob/master/hadoop-ozone/dev-support/checks/coverage.sh) merges coverage when earlier jobs produced it. + +Some jobs use a **matrix** (for example multiple Java versions) with **fail-fast**: if one matrix leg fails, the others in that matrix stop. Unrelated jobs can still run until they finish or fail. + +## Other workflows + +The [workflows directory](https://github.com/apache/ozone/tree/master/.github/workflows) also contains jobs for caches, labeling, Ratis builds, repeating tests, generated docs, and more. The folder on `master` is the up-to-date list. + +### Stale pull requests + +[`close-stale-prs.yaml`](https://github.com/apache/ozone/blob/master/.github/workflows/close-stale-prs.yaml) runs on a timer and uses [actions/stale](https://github.com/actions/stale) to nudge and eventually close very inactive PRs. Exact timings are in that file. + +## If something fails + +:::note Green CI is a team norm + +A red check does not mean you did something wrong—it means the run found something to fix. It happens to everyone. + +::: + +1. Open the failed **`build-branch`** run → click the red job → read the **log** from the bottom upward for the first error. +2. If the job uploaded **Artifacts**, download them from the run summary (they expire after a short time). +3. Try the same **check script** locally if you have the environment set up ([Run checks on your machine](#run-checks-on-your-machine)). +4. To re-run without new code, use **Re-run all jobs** or **Re-run failed jobs** on the run page, or: + +```bash +git commit --allow-empty -m 'trigger new CI check' +``` + +Failures on the **main** repo’s default branch sometimes leave extra artifacts on [ozone build results](https://elek.github.io/ozone-build-results/) (community mirror). + +### Get help + +- Ask on your **pull request**—reviewers can interpret unfamiliar failures quickly. +- **Email** [[email protected]](mailto:[email protected]) for broader questions. +- **[GitHub Discussions](https://github.com/apache/ozone/discussions)** and the [#ozone](http://s.apache.org/slack-invite) Slack channel (ASF Slack) are listed in [`CONTRIBUTING.md`](https://github.com/apache/ozone/blob/master/CONTRIBUTING.md#who-to-contact). + +### Local and wiki testing notes + +- Wiki: [Running Ozone smoke tests and unit tests](https://cwiki.apache.org/confluence/display/OZONE/Running+Ozone+Smoke+Tests+and+Unit+Tests). + +## Advanced: flaky tests and debugging on a fork + +These patterns are for **repeat failures** or **environment-only** bugs. They usually live on a **personal fork**, not in `apache/ozone`. + +- Wiki: [GitHub Actions tips and tricks](https://cwiki.apache.org/confluence/display/OZONE/Github+Actions+tips+and+tricks) — running one test many times, extra logging, optional [tmate](https://github.com/tmate-io/tmate)-style access (unsafe on public repos; never with secrets exposed). +- Prefer current runner images (for example `ubuntu-latest`) when copying older examples. Review Comment: I think this is outdated. We have the more user-friendly `flaky-test-check` and `repeat-acceptance-test` workflows. They should be used in one's own fork. ########## docs/08-developer-guide/03-test/04-continuous-integration.md: ########## @@ -1,15 +1,160 @@ --- -draft: true sidebar_label: Continuous Integration --- # Continuous Integration With GitHub Actions -**TODO:** File a subtask under [HDDS-9861](https://issues.apache.org/jira/browse/HDDS-9861) and complete this page or section. +If you are new to the project, **you do not need to understand every job** below on day one. The goal of this page is to help you get a green **`build-branch`** run on your fork, know where to look when something fails, and find deeper detail when you need it. -Aggregate content from our various GitHub actions guides, including +Apache Ozone uses [GitHub Actions](https://docs.github.com/en/actions) to build and test every meaningful change. Workflow files live in [`.github/workflows`](https://github.com/apache/ozone/tree/master/.github/workflows) in [`apache/ozone`](https://github.com/apache/ozone). A longer, file-by-file reference lives in [`.github/ci.md`](https://github.com/apache/ozone/blob/master/.github/ci.md). -- [ci.md](https://github.com/apache/ozone/blob/master/.github/ci.md) -- [CONTRIBUTING.md](https://github.com/apache/ozone/blob/master/CONTRIBUTING.md#check-your-contribution) -- https://cwiki.apache.org/confluence/display/OZONE/Ozone+CI+with+Github+Actions -- https://cwiki.apache.org/confluence/display/OZONE/Github+Actions+tips+and+tricks. +:::info Use the right repository + +This page is about **[`apache/ozone`](https://github.com/apache/ozone)** (the Ozone product source code). The documentation site you are reading comes from **[`apache/ozone-site`](https://github.com/apache/ozone-site)** and has its **own** CI. For website-only edits, use the [ozone-site contributing guide](https://github.com/apache/ozone-site/blob/master/CONTRIBUTING.md). + +::: + +## Start here: your first code contribution + +Follow these steps once; after that, pushing to your branch is the usual loop. + +1. **Fork and clone** [`apache/ozone`](https://github.com/apache/ozone) to **your** GitHub account, then clone **your fork** locally. You will push branches to `origin` on the fork, then open a PR to `apache/ozone`. +2. **Turn on Actions** on the fork so workflows actually run ([how to enable them](#enable-github-actions-on-your-fork)). +3. **Jira** — Create or choose an issue in [HDDS](https://issues.apache.org/jira/projects/HDDS/) (the Ozone Jira project; the name is historical). Need an account? Use the ASF [Jira self-service](https://selfserve.apache.org/jira-account.html?project=ozone) form. +4. **Branch** — Work on a branch, often named after the issue (for example `HDDS-1234`). +5. **Push** — When you push, GitHub should show a **`build-branch`** workflow run under the **Actions** tab on your fork. Wait for it to finish and fix any failures you can reproduce. +6. **Open the PR** — Use the [pull request template](https://github.com/apache/ozone/blob/master/.github/pull_request_template.md). When the change is ready for review, set the Jira to **Patch Available** so committers know to look. + +The full narrative (reviews, merging, Jira etiquette) is in the [Ozone contributing guide](https://github.com/apache/ozone/blob/master/CONTRIBUTING.md#contribute-your-modifications). + +:::tip You can lean on CI first + +Many contributors fix quick issues by reading the failing log on GitHub, then pushing a small follow-up commit. Running every check locally is **optional but helpful** for faster feedback; see [Run checks on your machine](#run-checks-on-your-machine). + +::: + +## Enable GitHub Actions on your fork + +New forks sometimes have workflows off until you allow them. + +1. Open **your fork** on GitHub → **Settings** → **Actions** → **General**. +2. Under **Actions permissions**, pick a policy that allows workflows to run (many people use **Allow all actions and reusable workflows** on personal forks). +3. Open the **Actions** tab. If GitHub asks to enable workflows, confirm so **`build-branch`** runs when you push. + +More detail: [Enabling or disabling GitHub Actions](https://docs.github.com/en/repositories/managing-your-repositorys-settings-and-features/enabling-or-disabling-github-actions-for-a-repository). + +## What you see in GitHub: `build-branch` + +Two names show up in docs; both mean “the main CI pipeline”: + +| What | Meaning | +| --- | --- | +| **`build-branch`** | The **name** of the workflow in the Actions tab. It comes from the `name:` field in [`post-commit.yml`](https://github.com/apache/ozone/blob/master/.github/workflows/post-commit.yml). | +| **`ci.yml`** | Where most **jobs** (compile, tests, and so on) are defined. `post-commit.yml` calls this file as a [reusable workflow](https://docs.github.com/en/actions/using-workflows/reusing-workflows). | + +So: **`post-commit.yml`** = front door; **`ci.yml`** = where the heavy lifting is described. + +When you push new commits to an open pull request, **newer runs can cancel older ones** still in progress ([concurrency](https://docs.github.com/en/actions/using-jobs/using-concurrency)). That is normal and saves time. + +## Run checks on your machine + +Running scripts locally catches problems before you push. You need a working dev environment first—see [Build with Maven](../../developer-guide/build/maven) and [Building from source](https://github.com/apache/ozone/blob/master/CONTRIBUTING.md#building-from-source) in `CONTRIBUTING.md`. + +From the **root of your clone** (the folder that contains `hadoop-ozone/`): + +```bash +./hadoop-ozone/dev-support/checks/build.sh +``` + +Most checks live in [`hadoop-ozone/dev-support/checks`](https://github.com/apache/ozone/tree/master/hadoop-ozone/dev-support/checks). The [Check your contribution](https://github.com/apache/ozone/blob/master/CONTRIBUTING.md#check-your-contribution) section groups them by rough duration: + +| Rough time | Scripts | What they do | +| --- | --- | --- | +| Build step | `build.sh` | Compile Ozone | +| Minutes | `author.sh`, `bats.sh`, `rat.sh`, `docs.sh`, `dependency.sh`, `checkstyle.sh`, `pmd.sh` | Style, license headers, docs, dependency list | +| ~10 minutes | `findbugs.sh`, `kubernetes.sh` | SpotBugs, small Kubernetes-related checks | +| An hour or more | `unit.sh`, `integration.sh`, `acceptance.sh` | Unit tests, mini-cluster tests, Docker Compose acceptance tests | + +More on test styles: [Acceptance tests](./acceptance-tests) on this site. For unit and integration testing (and running checks locally), see [Running Ozone smoke tests and unit tests](https://cwiki.apache.org/confluence/display/OZONE/Running+Ozone+Smoke+Tests+and+Unit+Tests) on the wiki until the dedicated **Unit tests** and **Integration tests** pages here are published. + +`integration.sh` and `acceptance.sh` can take extra arguments to run a subset; open the scripts to see options. Output usually lands under `target/` (for example `target/docs`). + +## Why did CI skip some jobs? + +Not every pull request runs every job. A step called **build-info** runs [`selective_ci_checks.sh`](https://github.com/apache/ozone/blob/master/dev-support/ci/selective_ci_checks.sh) and only enables jobs that match the files you changed—unless: + +- the run is **not** from a PR, or +- the PR has the **`full tests needed`** [label](https://docs.github.com/en/issues/using-labels-and-milestones-to-track-work/managing-labels). + +So a focused change might show fewer checks than a large refactor. **That is expected.** Reviewers can add **`full tests needed`** when the full matrix is required. If you think the wrong jobs were skipped, **ask on the PR**; reviewers are used to that question. + +## What the main CI jobs do (overview) + +The list below matches [`.github/ci.md`](https://github.com/apache/ozone/blob/master/.github/ci.md). Treat it as a map when reading logs, not something to memorize. + +- **build-info** — Decides which other jobs run (selective CI). +- **compile** — Builds with Java 8 and 11 via [`build.sh`](https://github.com/apache/ozone/blob/master/hadoop-ozone/dev-support/checks/build.sh); later jobs typically use the Java 8 build. +- **basic** — Checks like author tags, BATS, Checkstyle, Hugo for docs, SpotBugs, PMD, RAT—depending on what was selected. +- **unit** — [`unit.sh`](https://github.com/apache/ozone/blob/master/hadoop-ozone/dev-support/checks/unit.sh) and [`native.sh`](https://github.com/apache/ozone/blob/master/hadoop-ozone/dev-support/checks/native.sh) for RocksDB-native tests. +- **dependency** — Compares JARs to [`jar-report.txt`](https://github.com/apache/ozone/blob/master/hadoop-ozone/dist/src/main/license/jar-report.txt). +- **acceptance** — [`acceptance.sh`](https://github.com/apache/ozone/blob/master/hadoop-ozone/dev-support/checks/acceptance.sh) (Robot Framework + Docker Compose; variants like secure / unsecure / misc). +- **kubernetes** — [`kubernetes.sh`](https://github.com/apache/ozone/blob/master/hadoop-ozone/dev-support/checks/kubernetes.sh). +- **integration** — [`integration.sh`](https://github.com/apache/ozone/blob/master/hadoop-ozone/dev-support/checks/integration.sh) (mini-cluster style tests, often sharded in CI). +- **coverage** — [`coverage.sh`](https://github.com/apache/ozone/blob/master/hadoop-ozone/dev-support/checks/coverage.sh) merges coverage when earlier jobs produced it. + +Some jobs use a **matrix** (for example multiple Java versions) with **fail-fast**: if one matrix leg fails, the others in that matrix stop. Unrelated jobs can still run until they finish or fail. + +## Other workflows + +The [workflows directory](https://github.com/apache/ozone/tree/master/.github/workflows) also contains jobs for caches, labeling, Ratis builds, repeating tests, generated docs, and more. The folder on `master` is the up-to-date list. + +### Stale pull requests + +[`close-stale-prs.yaml`](https://github.com/apache/ozone/blob/master/.github/workflows/close-stale-prs.yaml) runs on a timer and uses [actions/stale](https://github.com/actions/stale) to nudge and eventually close very inactive PRs. Exact timings are in that file. + +## If something fails + +:::note Green CI is a team norm + +A red check does not mean you did something wrong—it means the run found something to fix. It happens to everyone. + +::: + +1. Open the failed **`build-branch`** run → click the red job → read the **log** from the bottom upward for the first error. +2. If the job uploaded **Artifacts**, download them from the run summary (they expire after a short time). Review Comment: I don't think expiry needs to be mentioned here. Build artifacts (binary and source tarballs) expire after 1 day, but test results only after 90 days (the default on GitHub). ########## docs/08-developer-guide/03-test/04-continuous-integration.md: ########## @@ -1,15 +1,160 @@ --- -draft: true sidebar_label: Continuous Integration --- # Continuous Integration With GitHub Actions -**TODO:** File a subtask under [HDDS-9861](https://issues.apache.org/jira/browse/HDDS-9861) and complete this page or section. +If you are new to the project, **you do not need to understand every job** below on day one. The goal of this page is to help you get a green **`build-branch`** run on your fork, know where to look when something fails, and find deeper detail when you need it. -Aggregate content from our various GitHub actions guides, including +Apache Ozone uses [GitHub Actions](https://docs.github.com/en/actions) to build and test every meaningful change. Workflow files live in [`.github/workflows`](https://github.com/apache/ozone/tree/master/.github/workflows) in [`apache/ozone`](https://github.com/apache/ozone). A longer, file-by-file reference lives in [`.github/ci.md`](https://github.com/apache/ozone/blob/master/.github/ci.md). -- [ci.md](https://github.com/apache/ozone/blob/master/.github/ci.md) -- [CONTRIBUTING.md](https://github.com/apache/ozone/blob/master/CONTRIBUTING.md#check-your-contribution) -- https://cwiki.apache.org/confluence/display/OZONE/Ozone+CI+with+Github+Actions -- https://cwiki.apache.org/confluence/display/OZONE/Github+Actions+tips+and+tricks. +:::info Use the right repository + +This page is about **[`apache/ozone`](https://github.com/apache/ozone)** (the Ozone product source code). The documentation site you are reading comes from **[`apache/ozone-site`](https://github.com/apache/ozone-site)** and has its **own** CI. For website-only edits, use the [ozone-site contributing guide](https://github.com/apache/ozone-site/blob/master/CONTRIBUTING.md). + +::: + +## Start here: your first code contribution + +Follow these steps once; after that, pushing to your branch is the usual loop. + +1. **Fork and clone** [`apache/ozone`](https://github.com/apache/ozone) to **your** GitHub account, then clone **your fork** locally. You will push branches to `origin` on the fork, then open a PR to `apache/ozone`. +2. **Turn on Actions** on the fork so workflows actually run ([how to enable them](#enable-github-actions-on-your-fork)). +3. **Jira** — Create or choose an issue in [HDDS](https://issues.apache.org/jira/projects/HDDS/) (the Ozone Jira project; the name is historical). Need an account? Use the ASF [Jira self-service](https://selfserve.apache.org/jira-account.html?project=ozone) form. +4. **Branch** — Work on a branch, often named after the issue (for example `HDDS-1234`). +5. **Push** — When you push, GitHub should show a **`build-branch`** workflow run under the **Actions** tab on your fork. Wait for it to finish and fix any failures you can reproduce. +6. **Open the PR** — Use the [pull request template](https://github.com/apache/ozone/blob/master/.github/pull_request_template.md). When the change is ready for review, set the Jira to **Patch Available** so committers know to look. + +The full narrative (reviews, merging, Jira etiquette) is in the [Ozone contributing guide](https://github.com/apache/ozone/blob/master/CONTRIBUTING.md#contribute-your-modifications). + +:::tip You can lean on CI first + +Many contributors fix quick issues by reading the failing log on GitHub, then pushing a small follow-up commit. Running every check locally is **optional but helpful** for faster feedback; see [Run checks on your machine](#run-checks-on-your-machine). + +::: + +## Enable GitHub Actions on your fork + +New forks sometimes have workflows off until you allow them. + +1. Open **your fork** on GitHub → **Settings** → **Actions** → **General**. +2. Under **Actions permissions**, pick a policy that allows workflows to run (many people use **Allow all actions and reusable workflows** on personal forks). +3. Open the **Actions** tab. If GitHub asks to enable workflows, confirm so **`build-branch`** runs when you push. + +More detail: [Enabling or disabling GitHub Actions](https://docs.github.com/en/repositories/managing-your-repositorys-settings-and-features/enabling-or-disabling-github-actions-for-a-repository). + +## What you see in GitHub: `build-branch` + +Two names show up in docs; both mean “the main CI pipeline”: + +| What | Meaning | +| --- | --- | +| **`build-branch`** | The **name** of the workflow in the Actions tab. It comes from the `name:` field in [`post-commit.yml`](https://github.com/apache/ozone/blob/master/.github/workflows/post-commit.yml). | +| **`ci.yml`** | Where most **jobs** (compile, tests, and so on) are defined. `post-commit.yml` calls this file as a [reusable workflow](https://docs.github.com/en/actions/using-workflows/reusing-workflows). | + +So: **`post-commit.yml`** = front door; **`ci.yml`** = where the heavy lifting is described. + +When you push new commits to an open pull request, **newer runs can cancel older ones** still in progress ([concurrency](https://docs.github.com/en/actions/using-jobs/using-concurrency)). That is normal and saves time. + +## Run checks on your machine + +Running scripts locally catches problems before you push. You need a working dev environment first—see [Build with Maven](../../developer-guide/build/maven) and [Building from source](https://github.com/apache/ozone/blob/master/CONTRIBUTING.md#building-from-source) in `CONTRIBUTING.md`. + +From the **root of your clone** (the folder that contains `hadoop-ozone/`): + +```bash +./hadoop-ozone/dev-support/checks/build.sh +``` + +Most checks live in [`hadoop-ozone/dev-support/checks`](https://github.com/apache/ozone/tree/master/hadoop-ozone/dev-support/checks). The [Check your contribution](https://github.com/apache/ozone/blob/master/CONTRIBUTING.md#check-your-contribution) section groups them by rough duration: + +| Rough time | Scripts | What they do | +| --- | --- | --- | +| Build step | `build.sh` | Compile Ozone | +| Minutes | `author.sh`, `bats.sh`, `rat.sh`, `docs.sh`, `dependency.sh`, `checkstyle.sh`, `pmd.sh` | Style, license headers, docs, dependency list | +| ~10 minutes | `findbugs.sh`, `kubernetes.sh` | SpotBugs, small Kubernetes-related checks | +| An hour or more | `unit.sh`, `integration.sh`, `acceptance.sh` | Unit tests, mini-cluster tests, Docker Compose acceptance tests | + +More on test styles: [Acceptance tests](./acceptance-tests) on this site. For unit and integration testing (and running checks locally), see [Running Ozone smoke tests and unit tests](https://cwiki.apache.org/confluence/display/OZONE/Running+Ozone+Smoke+Tests+and+Unit+Tests) on the wiki until the dedicated **Unit tests** and **Integration tests** pages here are published. + +`integration.sh` and `acceptance.sh` can take extra arguments to run a subset; open the scripts to see options. Output usually lands under `target/` (for example `target/docs`). + +## Why did CI skip some jobs? + +Not every pull request runs every job. A step called **build-info** runs [`selective_ci_checks.sh`](https://github.com/apache/ozone/blob/master/dev-support/ci/selective_ci_checks.sh) and only enables jobs that match the files you changed—unless: + +- the run is **not** from a PR, or +- the PR has the **`full tests needed`** [label](https://docs.github.com/en/issues/using-labels-and-milestones-to-track-work/managing-labels). + +So a focused change might show fewer checks than a large refactor. **That is expected.** Reviewers can add **`full tests needed`** when the full matrix is required. If you think the wrong jobs were skipped, **ask on the PR**; reviewers are used to that question. + +## What the main CI jobs do (overview) + +The list below matches [`.github/ci.md`](https://github.com/apache/ozone/blob/master/.github/ci.md). Treat it as a map when reading logs, not something to memorize. + +- **build-info** — Decides which other jobs run (selective CI). +- **compile** — Builds with Java 8 and 11 via [`build.sh`](https://github.com/apache/ozone/blob/master/hadoop-ozone/dev-support/checks/build.sh); later jobs typically use the Java 8 build. +- **basic** — Checks like author tags, BATS, Checkstyle, Hugo for docs, SpotBugs, PMD, RAT—depending on what was selected. +- **unit** — [`unit.sh`](https://github.com/apache/ozone/blob/master/hadoop-ozone/dev-support/checks/unit.sh) and [`native.sh`](https://github.com/apache/ozone/blob/master/hadoop-ozone/dev-support/checks/native.sh) for RocksDB-native tests. +- **dependency** — Compares JARs to [`jar-report.txt`](https://github.com/apache/ozone/blob/master/hadoop-ozone/dist/src/main/license/jar-report.txt). +- **acceptance** — [`acceptance.sh`](https://github.com/apache/ozone/blob/master/hadoop-ozone/dev-support/checks/acceptance.sh) (Robot Framework + Docker Compose; variants like secure / unsecure / misc). +- **kubernetes** — [`kubernetes.sh`](https://github.com/apache/ozone/blob/master/hadoop-ozone/dev-support/checks/kubernetes.sh). +- **integration** — [`integration.sh`](https://github.com/apache/ozone/blob/master/hadoop-ozone/dev-support/checks/integration.sh) (mini-cluster style tests, often sharded in CI). +- **coverage** — [`coverage.sh`](https://github.com/apache/ozone/blob/master/hadoop-ozone/dev-support/checks/coverage.sh) merges coverage when earlier jobs produced it. + +Some jobs use a **matrix** (for example multiple Java versions) with **fail-fast**: if one matrix leg fails, the others in that matrix stop. Unrelated jobs can still run until they finish or fail. + +## Other workflows + +The [workflows directory](https://github.com/apache/ozone/tree/master/.github/workflows) also contains jobs for caches, labeling, Ratis builds, repeating tests, generated docs, and more. The folder on `master` is the up-to-date list. + +### Stale pull requests + +[`close-stale-prs.yaml`](https://github.com/apache/ozone/blob/master/.github/workflows/close-stale-prs.yaml) runs on a timer and uses [actions/stale](https://github.com/actions/stale) to nudge and eventually close very inactive PRs. Exact timings are in that file. + +## If something fails + +:::note Green CI is a team norm + +A red check does not mean you did something wrong—it means the run found something to fix. It happens to everyone. + +::: + +1. Open the failed **`build-branch`** run → click the red job → read the **log** from the bottom upward for the first error. +2. If the job uploaded **Artifacts**, download them from the run summary (they expire after a short time). +3. Try the same **check script** locally if you have the environment set up ([Run checks on your machine](#run-checks-on-your-machine)). +4. To re-run without new code, use **Re-run all jobs** or **Re-run failed jobs** on the run page, or: + +```bash +git commit --allow-empty -m 'trigger new CI check' +``` + +Failures on the **main** repo’s default branch sometimes leave extra artifacts on [ozone build results](https://elek.github.io/ozone-build-results/) (community mirror). + +### Get help + +- Ask on your **pull request**—reviewers can interpret unfamiliar failures quickly. +- **Email** [[email protected]](mailto:[email protected]) for broader questions. +- **[GitHub Discussions](https://github.com/apache/ozone/discussions)** and the [#ozone](http://s.apache.org/slack-invite) Slack channel (ASF Slack) are listed in [`CONTRIBUTING.md`](https://github.com/apache/ozone/blob/master/CONTRIBUTING.md#who-to-contact). + +### Local and wiki testing notes + +- Wiki: [Running Ozone smoke tests and unit tests](https://cwiki.apache.org/confluence/display/OZONE/Running+Ozone+Smoke+Tests+and+Unit+Tests). + +## Advanced: flaky tests and debugging on a fork + +These patterns are for **repeat failures** or **environment-only** bugs. They usually live on a **personal fork**, not in `apache/ozone`. + +- Wiki: [GitHub Actions tips and tricks](https://cwiki.apache.org/confluence/display/OZONE/Github+Actions+tips+and+tricks) — running one test many times, extra logging, optional [tmate](https://github.com/tmate-io/tmate)-style access (unsafe on public repos; never with secrets exposed). +- Prefer current runner images (for example `ubuntu-latest`) when copying older examples. + +## Deprecated workflows + +Old workflows can still appear on the [Actions](https://github.com/apache/ozone/actions) tab. An outdated workflow also named **build-branch** is tied to [`chaos.yml`](https://github.com/apache/ozone/actions/workflows/chaos.yml), not the current `post-commit.yml` pipeline—compare URLs. Full list: [`.github/ci.md` — Old/Deprecated Workflows](https://github.com/apache/ozone/blob/master/.github/ci.md#olddeprecated-workflows). + +## See also + +- [`.github/ci.md`](https://github.com/apache/ozone/blob/master/.github/ci.md) in `apache/ozone` +- [Contributing guide — Check your contribution](https://github.com/apache/ozone/blob/master/CONTRIBUTING.md#check-your-contribution) +- Wiki: [Ozone CI with GitHub Actions](https://cwiki.apache.org/confluence/display/OZONE/Ozone+CI+with+Github+Actions) (older context; prefer the repo for current names) +- Wiki: [GitHub Actions tips and tricks](https://cwiki.apache.org/confluence/display/OZONE/Github+Actions+tips+and+tricks) Review Comment: I would omit these links to outdated docs that we are trying to migrate content from. (Contributing guide is not outdated, but there is another link to it above.) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
