adoroszlai commented on code in PR #382: URL: https://github.com/apache/ozone-site/pull/382#discussion_r3081351667
########## docs/08-developer-guide/03-test/04-continuous-integration.md: ########## @@ -1,15 +1,161 @@ --- -draft: true sidebar_label: Continuous Integration --- # Continuous Integration With GitHub Actions -**TODO:** File a subtask under [HDDS-9861](https://issues.apache.org/jira/browse/HDDS-9861) and complete this page or section. +If you are new to the project, **you do not need to understand every job** below on day one. The goal of this page is to help you get a green **`build-branch`** run on your fork, know where to look when something fails, and find deeper detail when you need it. -Aggregate content from our various GitHub actions guides, including +Apache Ozone uses [GitHub Actions](https://docs.github.com/en/actions) to build and test every meaningful change. Workflow files live in [`.github/workflows`](https://github.com/apache/ozone/tree/master/.github/workflows) in [`apache/ozone`](https://github.com/apache/ozone). A longer, file-by-file reference lives in [`.github/ci.md`](https://github.com/apache/ozone/blob/master/.github/ci.md). -- [ci.md](https://github.com/apache/ozone/blob/master/.github/ci.md) -- [CONTRIBUTING.md](https://github.com/apache/ozone/blob/master/CONTRIBUTING.md#check-your-contribution) -- https://cwiki.apache.org/confluence/display/OZONE/Ozone+CI+with+Github+Actions -- https://cwiki.apache.org/confluence/display/OZONE/Github+Actions+tips+and+tricks. +:::info Use the right repository + +This page is about **[`apache/ozone`](https://github.com/apache/ozone)** (the Ozone product source code). The documentation site you are reading comes from **[`apache/ozone-site`](https://github.com/apache/ozone-site)** and has its **own** CI. For website-only edits, use the [ozone-site contributing guide](https://github.com/apache/ozone-site/blob/master/CONTRIBUTING.md). + +::: + +## Start here: your first code contribution + +Follow these steps once; after that, pushing to your branch is the usual loop. + +1. **Fork and clone** [`apache/ozone`](https://github.com/apache/ozone) to **your** GitHub account, then clone **your fork** locally. You will push branches to `origin` on the fork, then open a PR to `apache/ozone`. +2. **Turn on Actions** on the fork so workflows actually run ([how to enable them](#enable-github-actions-on-your-fork)). +3. **Jira** — Create or choose an issue in [HDDS](https://issues.apache.org/jira/projects/HDDS/) (the Ozone Jira project; the name is historical). Need an account? Use the ASF [Jira self-service](https://selfserve.apache.org/jira-account.html?project=ozone) form. +4. **Branch** — Work on a branch, often named after the issue (for example `HDDS-1234`). +5. **Push** — When you push, GitHub should show a **`build-branch`** workflow run under the **Actions** tab on your fork. Wait for it to finish and fix any failures you can reproduce. +6. **Open the PR** — Use the [pull request template](https://github.com/apache/ozone/blob/master/.github/pull_request_template.md). When the change is ready for review, set the Jira to **Patch Available** so committers know to look. + +The full narrative (reviews, merging, Jira etiquette) is in the [Ozone contributing guide](https://github.com/apache/ozone/blob/master/CONTRIBUTING.md#contribute-your-modifications). + +:::tip You can lean on CI first + +Many contributors fix quick issues by reading the failing log on GitHub, then pushing a small follow-up commit. Running every check locally is **optional but helpful** for faster feedback; see [Run checks on your machine](#run-checks-on-your-machine). + +::: + +## Enable GitHub Actions on your fork + +New forks sometimes have workflows off until you allow them. + +1. Open **your fork** on GitHub → **Settings** → **Actions** → **General**. +2. Under **Actions permissions**, pick a policy that allows workflows to run (many people use **Allow all actions and reusable workflows** on personal forks). +3. Open the **Actions** tab. If GitHub asks to enable workflows, confirm so **`build-branch`** runs when you push. + +More detail: [Enabling or disabling GitHub Actions](https://docs.github.com/en/repositories/managing-your-repositorys-settings-and-features/enabling-or-disabling-github-actions-for-a-repository). + +## What you see in GitHub: `build-branch` + +Two names show up in docs; both mean “the main CI pipeline”: + +| What | Meaning | +| --- | --- | +| **`build-branch`** | The **name** of the workflow in the Actions tab. It comes from the `name:` field in [`post-commit.yml`](https://github.com/apache/ozone/blob/master/.github/workflows/post-commit.yml). | +| **`ci.yml`** | Where most **jobs** (build, compile, tests, and so on) are defined. `post-commit.yml` calls this file as a [reusable workflow](https://docs.github.com/en/actions/using-workflows/reusing-workflows). | + +So: **`post-commit.yml`** = front door; **`ci.yml`** = where the heavy lifting is described. + +When you push new commits to an open pull request, **newer runs can cancel older ones** still in progress ([concurrency](https://docs.github.com/en/actions/using-jobs/using-concurrency)). That is normal and saves time. + +## Run checks on your machine + +Running scripts locally catches problems before you push. You need a working dev environment first—see [Build with Maven](../../developer-guide/build/maven) and [Building from source](https://github.com/apache/ozone/blob/master/CONTRIBUTING.md#building-from-source) in `CONTRIBUTING.md`. + +Most checks live in [`hadoop-ozone/dev-support/checks`](https://github.com/apache/ozone/tree/master/hadoop-ozone/dev-support/checks). The [Check your contribution](https://github.com/apache/ozone/blob/master/CONTRIBUTING.md#check-your-contribution) section groups them by rough duration: + +| Rough time | Scripts | What they do | +| --- | --- | --- | +| Build step | `build.sh` | Compile Ozone | +| Minutes | `author.sh`, `bats.sh`, `rat.sh`, `docs.sh`, `dependency.sh`, `checkstyle.sh`, `pmd.sh` | Style, license headers, docs, dependency list | +| ~10 minutes | `findbugs.sh`, `kubernetes.sh` | SpotBugs, small Kubernetes-related checks | +| An hour or more | `integration.sh`, `acceptance.sh` | JUnit tests (via integration), mini-cluster style tests, Docker Compose acceptance tests | + +The command below is **only an example** of running one script from the **root of your clone** (the folder that contains `hadoop-ozone/`): + +```bash +./hadoop-ozone/dev-support/checks/build.sh +``` + +More on test styles: [Acceptance tests](./acceptance-tests) on this site. + +### Reproducing failures locally (by check type) + +What you need depends on which job failed: + +1. **basic** — Safe to run locally without a full prior build; scripts are quick (author tags, BATS, RAT, docs, Checkstyle, PMD, SpotBugs, and similar—whatever `basic` selected for that run). +2. **dependency / license** — Quick, but expects a **build** already (the dependency check compares against built outputs / the dependency list). +3. **Checks that reproduce compiler or packaging issues** — Run the same **`build.sh`** (or narrower Maven command) after a normal dev build; align with the log. +4. **integration** (JUnit) — After a build, narrow work to one test with Maven, for example `-Dtest='YourTestClass'`, or run the same class or method from your IDE. [`integration.sh`](https://github.com/apache/ozone/blob/master/hadoop-ozone/dev-support/checks/integration.sh) wraps the full suite; open it for flags and defaults. +5. **acceptance** — Needs a **build** (often a dist build). Prefer re-running only the failing shell driver the log names (for example a line like `ERROR: Test execution of ozone/test-legacy-bucket.sh is FAILED` points at `test-legacy-bucket.sh`) instead of the whole suite. +6. **kubernetes** — The `kubernetes.sh` check is aimed at **Linux** environments; macOS or Windows may not match CI. + +`integration.sh` and `acceptance.sh` can take extra arguments to run a subset; open the scripts to see options. Output usually lands under `target/` (for example `target/docs`). + +## Why did CI skip some jobs? + +Not every pull request runs every job. A step called **build-info** runs [`selective_ci_checks.sh`](https://github.com/apache/ozone/blob/master/dev-support/ci/selective_ci_checks.sh) and only enables jobs that match the files you changed—unless: + +- the run is **not** from a PR, or +- the PR has the **`full tests needed`** [label](https://docs.github.com/en/issues/using-labels-and-milestones-to-track-work/managing-labels). + +So a focused change might show fewer checks than a large refactor. **That is expected.** Reviewers can add **`full tests needed`** when the full matrix is required. If you think the wrong jobs were skipped, **ask on the PR**; reviewers are used to that question. + +## What the main CI jobs do (overview) + +The list below matches [`.github/ci.md`](https://github.com/apache/ozone/blob/master/.github/ci.md). Treat it as a map when reading logs, not something to memorize. + +- **build-info** — Decides which other jobs run (selective CI). +- **build** — Performs a full build; its output is reused by later jobs. +- **compile** — Re-builds with various Java versions. It consumes the **source tarball** (release artifact) produced by **build**, not a fresh checkout of the git repository. +- **basic** — Checks like author tags, BATS, Checkstyle, Hugo for docs, SpotBugs, PMD, RAT—depending on what was selected. +- **dependency** — Detects whether dependencies were added or removed, as a reminder to update `LICENSE.txt` (how this is implemented—for example comparisons against `jar-report.txt`—is an internal detail). Review Comment: "how this is implemented ..." was intended as a rationale for the suggestion, not to be included in the doc. ########## docs/08-developer-guide/03-test/04-continuous-integration.md: ########## @@ -1,15 +1,161 @@ --- -draft: true sidebar_label: Continuous Integration --- # Continuous Integration With GitHub Actions -**TODO:** File a subtask under [HDDS-9861](https://issues.apache.org/jira/browse/HDDS-9861) and complete this page or section. +If you are new to the project, **you do not need to understand every job** below on day one. The goal of this page is to help you get a green **`build-branch`** run on your fork, know where to look when something fails, and find deeper detail when you need it. -Aggregate content from our various GitHub actions guides, including +Apache Ozone uses [GitHub Actions](https://docs.github.com/en/actions) to build and test every meaningful change. Workflow files live in [`.github/workflows`](https://github.com/apache/ozone/tree/master/.github/workflows) in [`apache/ozone`](https://github.com/apache/ozone). A longer, file-by-file reference lives in [`.github/ci.md`](https://github.com/apache/ozone/blob/master/.github/ci.md). -- [ci.md](https://github.com/apache/ozone/blob/master/.github/ci.md) -- [CONTRIBUTING.md](https://github.com/apache/ozone/blob/master/CONTRIBUTING.md#check-your-contribution) -- https://cwiki.apache.org/confluence/display/OZONE/Ozone+CI+with+Github+Actions -- https://cwiki.apache.org/confluence/display/OZONE/Github+Actions+tips+and+tricks. +:::info Use the right repository + +This page is about **[`apache/ozone`](https://github.com/apache/ozone)** (the Ozone product source code). The documentation site you are reading comes from **[`apache/ozone-site`](https://github.com/apache/ozone-site)** and has its **own** CI. For website-only edits, use the [ozone-site contributing guide](https://github.com/apache/ozone-site/blob/master/CONTRIBUTING.md). + +::: + +## Start here: your first code contribution + +Follow these steps once; after that, pushing to your branch is the usual loop. + +1. **Fork and clone** [`apache/ozone`](https://github.com/apache/ozone) to **your** GitHub account, then clone **your fork** locally. You will push branches to `origin` on the fork, then open a PR to `apache/ozone`. +2. **Turn on Actions** on the fork so workflows actually run ([how to enable them](#enable-github-actions-on-your-fork)). +3. **Jira** — Create or choose an issue in [HDDS](https://issues.apache.org/jira/projects/HDDS/) (the Ozone Jira project; the name is historical). Need an account? Use the ASF [Jira self-service](https://selfserve.apache.org/jira-account.html?project=ozone) form. +4. **Branch** — Work on a branch, often named after the issue (for example `HDDS-1234`). +5. **Push** — When you push, GitHub should show a **`build-branch`** workflow run under the **Actions** tab on your fork. Wait for it to finish and fix any failures you can reproduce. +6. **Open the PR** — Use the [pull request template](https://github.com/apache/ozone/blob/master/.github/pull_request_template.md). When the change is ready for review, set the Jira to **Patch Available** so committers know to look. + +The full narrative (reviews, merging, Jira etiquette) is in the [Ozone contributing guide](https://github.com/apache/ozone/blob/master/CONTRIBUTING.md#contribute-your-modifications). + +:::tip You can lean on CI first + +Many contributors fix quick issues by reading the failing log on GitHub, then pushing a small follow-up commit. Running every check locally is **optional but helpful** for faster feedback; see [Run checks on your machine](#run-checks-on-your-machine). + +::: + +## Enable GitHub Actions on your fork + +New forks sometimes have workflows off until you allow them. + +1. Open **your fork** on GitHub → **Settings** → **Actions** → **General**. +2. Under **Actions permissions**, pick a policy that allows workflows to run (many people use **Allow all actions and reusable workflows** on personal forks). +3. Open the **Actions** tab. If GitHub asks to enable workflows, confirm so **`build-branch`** runs when you push. + +More detail: [Enabling or disabling GitHub Actions](https://docs.github.com/en/repositories/managing-your-repositorys-settings-and-features/enabling-or-disabling-github-actions-for-a-repository). + +## What you see in GitHub: `build-branch` + +Two names show up in docs; both mean “the main CI pipeline”: + +| What | Meaning | +| --- | --- | +| **`build-branch`** | The **name** of the workflow in the Actions tab. It comes from the `name:` field in [`post-commit.yml`](https://github.com/apache/ozone/blob/master/.github/workflows/post-commit.yml). | +| **`ci.yml`** | Where most **jobs** (build, compile, tests, and so on) are defined. `post-commit.yml` calls this file as a [reusable workflow](https://docs.github.com/en/actions/using-workflows/reusing-workflows). | + +So: **`post-commit.yml`** = front door; **`ci.yml`** = where the heavy lifting is described. + +When you push new commits to an open pull request, **newer runs can cancel older ones** still in progress ([concurrency](https://docs.github.com/en/actions/using-jobs/using-concurrency)). That is normal and saves time. + +## Run checks on your machine + +Running scripts locally catches problems before you push. You need a working dev environment first—see [Build with Maven](../../developer-guide/build/maven) and [Building from source](https://github.com/apache/ozone/blob/master/CONTRIBUTING.md#building-from-source) in `CONTRIBUTING.md`. + +Most checks live in [`hadoop-ozone/dev-support/checks`](https://github.com/apache/ozone/tree/master/hadoop-ozone/dev-support/checks). The [Check your contribution](https://github.com/apache/ozone/blob/master/CONTRIBUTING.md#check-your-contribution) section groups them by rough duration: + +| Rough time | Scripts | What they do | +| --- | --- | --- | +| Build step | `build.sh` | Compile Ozone | +| Minutes | `author.sh`, `bats.sh`, `rat.sh`, `docs.sh`, `dependency.sh`, `checkstyle.sh`, `pmd.sh` | Style, license headers, docs, dependency list | +| ~10 minutes | `findbugs.sh`, `kubernetes.sh` | SpotBugs, small Kubernetes-related checks | +| An hour or more | `integration.sh`, `acceptance.sh` | JUnit tests (via integration), mini-cluster style tests, Docker Compose acceptance tests | + +The command below is **only an example** of running one script from the **root of your clone** (the folder that contains `hadoop-ozone/`): + +```bash +./hadoop-ozone/dev-support/checks/build.sh +``` + +More on test styles: [Acceptance tests](./acceptance-tests) on this site. + +### Reproducing failures locally (by check type) + +What you need depends on which job failed: + +1. **basic** — Safe to run locally without a full prior build; scripts are quick (author tags, BATS, RAT, docs, Checkstyle, PMD, SpotBugs, and similar—whatever `basic` selected for that run). +2. **dependency / license** — Quick, but expects a **build** already (the dependency check compares against built outputs / the dependency list). +3. **Checks that reproduce compiler or packaging issues** — Run the same **`build.sh`** (or narrower Maven command) after a normal dev build; align with the log. +4. **integration** (JUnit) — After a build, narrow work to one test with Maven, for example `-Dtest='YourTestClass'`, or run the same class or method from your IDE. [`integration.sh`](https://github.com/apache/ozone/blob/master/hadoop-ozone/dev-support/checks/integration.sh) wraps the full suite; open it for flags and defaults. +5. **acceptance** — Needs a **build** (often a dist build). Prefer re-running only the failing shell driver the log names (for example a line like `ERROR: Test execution of ozone/test-legacy-bucket.sh is FAILED` points at `test-legacy-bucket.sh`) instead of the whole suite. +6. **kubernetes** — The `kubernetes.sh` check is aimed at **Linux** environments; macOS or Windows may not match CI. + +`integration.sh` and `acceptance.sh` can take extra arguments to run a subset; open the scripts to see options. Output usually lands under `target/` (for example `target/docs`). + +## Why did CI skip some jobs? + +Not every pull request runs every job. A step called **build-info** runs [`selective_ci_checks.sh`](https://github.com/apache/ozone/blob/master/dev-support/ci/selective_ci_checks.sh) and only enables jobs that match the files you changed—unless: + +- the run is **not** from a PR, or +- the PR has the **`full tests needed`** [label](https://docs.github.com/en/issues/using-labels-and-milestones-to-track-work/managing-labels). + +So a focused change might show fewer checks than a large refactor. **That is expected.** Reviewers can add **`full tests needed`** when the full matrix is required. If you think the wrong jobs were skipped, **ask on the PR**; reviewers are used to that question. + +## What the main CI jobs do (overview) + +The list below matches [`.github/ci.md`](https://github.com/apache/ozone/blob/master/.github/ci.md). Treat it as a map when reading logs, not something to memorize. + +- **build-info** — Decides which other jobs run (selective CI). +- **build** — Performs a full build; its output is reused by later jobs. +- **compile** — Re-builds with various Java versions. It consumes the **source tarball** (release artifact) produced by **build**, not a fresh checkout of the git repository. +- **basic** — Checks like author tags, BATS, Checkstyle, Hugo for docs, SpotBugs, PMD, RAT—depending on what was selected. +- **dependency** — Detects whether dependencies were added or removed, as a reminder to update `LICENSE.txt` (how this is implemented—for example comparisons against `jar-report.txt`—is an internal detail). Review Comment: Agree. "how this is implemented ..." was intended as a rationale for the suggestion, not to be included in the doc. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
