shahar1 opened a new pull request, #74889: URL: https://github.com/apache/airflow/pull/74889
Moves the trusted part of `Tests (AMD)` / `Tests (ARM)` onto self-hosted [RunsOn](https://runs-on.com) EC2 runners in the Airflow AWS account. Fork PRs stay on GitHub-hosted runners. This builds on the experiment in #74714 (`dont-merge`), which ran the whole CI on RunsOn and came back green. ## Which runs go where `breeze ci selective-check` now outputs `runner-type`, and build-info passes it on instead of hardcoding `["ubuntu-22.04"]` / `["ubuntu-22.04-arm"]`. The first matching rule wins: | Run | Runners | |---|---| | Any event in a repository other than `apache/airflow` (forks' own CI) | GitHub-hosted | | PR labelled `use public runners` | GitHub-hosted | | `push` / `schedule` / `workflow_dispatch` in `apache/airflow` (canary and release-branch pushes) | RunsOn | | Any other event except `pull_request` | GitHub-hosted | | PR labelled `use self-hosted runners`, if a committer (`COMMITTERS` in breeze) added it last and nobody removed it since | RunsOn | | PR labelled `non committer build` | GitHub-hosted | | PR opened by a bot (Dependabot, …) or from a `dependabot/` branch | GitHub-hosted | | PR from an `apache/airflow` branch | RunsOn | | PR from a fork | GitHub-hosted | To find out who added the label, selective checks read the PR's issue events. If they cannot be read, the label is ignored. Labels come from the triggering event, so a label change applies from the next push or reopen. Re-running an earlier run keeps its old labels, and "Re-run failed jobs" also keeps that run's runner decision. Only users with write access can re-run. `amd-runners` / `arm-runners` are unchanged and stay public. The Docker Hub release workflow consumes them, and it runs with release secrets. On RunsOn, `runner-type` is a label template (`runs-on=<run_id>/family=…/cpu={0}/ram={1}`). Each job picks its size class with `format(<label>, cpu, ram)`: 4/16, 8/32 or 32/128. On a public label the same call returns the label unchanged, so one workflow serves both paths. build-info, print-platform, notify-slack and the Windows smoke test keep their hardcoded GitHub-hosted labels. Steps that only make sense on RunsOn check `runner.environment == 'self-hosted'`: - **Image stashes**: on RunsOn, CI and prod image tarballs go to the stack's in-region S3 cache bucket under their sha256. The digest is stashed as a GitHub artifact with the ASF action, whose restore only accepts artifacts from the same run, or from the same branch and head repository (base branch for PRs). A tarball is extracted only if it matches that digest. The small caches (mount cache, prek, docs inventory) keep the ASF stash action on both paths. `ci-image-build` loads the stash actions from a `workflow-version/` checkout, because it also builds release refs that predate them. - **Cost summary**: `runs-on/action` v2.5.0 (ASF-allowlisted, no expiry) adds the job's EC2 cost to the job summary. ## Security The CI workflows trigger only on `pull_request`, `push`, `schedule` and `workflow_dispatch`. No workflow runs on `pull_request_target`, and this PR adds none. | Path | Event | Runner | Token / secrets | |---|---|---|---| | Fork PR | `pull_request` | GitHub-hosted | Read-only `GITHUB_TOKEN` (declared write permissions are downgraded), no secrets. Unchanged. | | Fork PR + committer's `use self-hosted runners` | `pull_request` | RunsOn | Same token and secrets as above, plus the EC2 instance role: read/write on the RunsOn cache bucket's `cache/` prefix. | | Dependabot PR | `pull_request` | GitHub-hosted | Read-only token, Dependabot secrets only. Unchanged. | | Dependabot PR + committer's `use self-hosted runners` | `pull_request` | RunsOn | Read-only token, Dependabot secrets only, plus the instance role. | | PR from an `apache/airflow` branch | `pull_request` | RunsOn | All declared permissions and repo secrets, as today, plus the instance role. Includes `packages: write` (image builds), `id-token: write` (image checks), `contents: write` (update-constraints, when dependencies are upgraded), `SLACK_BOT_TOKEN`, and `DOCS_AWS_*` passed to image checks (only canary steps use them). | | Canary | `push` / `schedule` / `workflow_dispatch` | RunsOn | As today. The jobs that publish docs (`DOCS_AWS_*`, `id-token: write`), update constraints (`contents: write`) and push the image cache (`packages: write`) now run on RunsOn. | - **What RunsOn adds.** Every job on RunsOn can write anywhere under `cache/` in the bucket. That is why image tarballs are content-addressed and checked against a GitHub-scoped digest. A PR's job can corrupt or delete a blob, which fails the job that needs it (denial of service). It cannot swap the image that a canary job with secrets restores. - **Routing decision.** The inputs are the base repository, event name, labels, the PR's head repository and author (from the event payload), and the issue events behind the label. A fork author controls none of those inputs. The `canary` label does not route to RunsOn. - **Not enforced by this PR.** Under `pull_request`, the code that evaluates those inputs belongs to the PR: a fork PR runs its own workflow files and breeze sources. It could therefore name RunsOn labels in `runs-on:` directly and skip selective checks entirely. The repository setting "Approval for running fork pull request workflows from contributors" only narrows this. Even **"Require approval for all external contributors"** exempts members of the `apache` GitHub organization (ASF committers of any project who linked their GitHub account) and repository collaborators (including the triage collaborators in `.asf.yaml`). So the real isolation has to come from the AWS / RunsOn side. Merge preconditions: - ASF Infra confirms the current approval setting. I could not read it with my token. - The instance role reaches only `cache/` in the cache bucket, with no access to RunsOn's config bucket. - The security group denies traffic between runners. Canary jobs bind an unauthenticated local registry and service ports on all interfaces. - The stack limits instance types and run time. - `S3CacheExpirationInDays` is at least 6, the longest digest retention. - A separate stack for labelled fork PRs is the stronger option. - **Labels persist.** Once a committer has labelled a fork PR, its later pushes also run on RunsOn (subject to the approval gate above). ## Attribution RunsOn's free license for open-source projects requires a public acknowledgement with a link to runs-on.com. This PR adds a text line to the README sponsors section. It adds no logo: the license reserves RunsOn's logos and grants no usage rights. ## Verification - `dev/breeze/tests/test_selective_checks.py`: 304 passed. New tests cover every routing rule, the label precedence, and the committer check on the label (committer, triager, re-added on a later page, unreadable events). - prek pre-commit hooks on the changed files pass, including yamllint, zizmor and `check-ci-workflows-in-sync`. - A local script walked all 24 reusable-workflow paths from ci-amd / ci-arm and confirmed every job formats the RunsOn template exactly once. - This PR comes from a fork branch. Its first CI run therefore goes to GitHub-hosted runners (the fork path). A second run, after a committer adds `use self-hosted runners`, goes to RunsOn and exercises the label path, including the provenance check. Results are below. The ARM RunsOn path (template, `ubuntu22-full-arm64` image, ASF stash on RunsOn) is only covered by a `workflow_dispatch` of `Tests (ARM)` or the first ARM canary. --- ##### Was generative AI tooling used to co-author this PR? - [X] Yes — Claude Code (Opus 5.5) Generated-by: Claude Code (Opus 5.5) following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions) --- * Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)** for more information. Note: commit author/co-author name and email in commits become permanently public when merged. * For fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed. * When adding dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). * For significant user-facing changes create newsfragment: `{pr_number}.significant.rst`, in [airflow-core/newsfragments](https://github.com/apache/airflow/tree/main/airflow-core/newsfragments). You can add this file in a follow-up commit after the PR is created so you know the PR number. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
