This is an automated email from the ASF dual-hosted git repository.
shahar1 pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow-ci-infra.git
The following commit(s) were added to refs/heads/main by this push:
new b34ae2d Retire the RunsOn CloudFormation stack (#87)
b34ae2d is described below
commit b34ae2dd7dec4d2e37b45f3eac8d26f3762cc044
Author: Shahar Epstein <[email protected]>
AuthorDate: Fri Nov 6 14:42:03 2026 +0200
Retire the RunsOn CloudFormation stack (#87)
Every job of apache/airflow has run on the runs-on-flex stack since
2026-11-01
(apache/airflow#76239), so the CloudFormation stack, the Terraform (RunsOn)
workflow and the
three roles that planned and applied it have no caller left. The stack
itself is destroyed by
hand before this merges: the apply role could only update it, never delete
it, and
runs-on-access is applied by hand anyway.
Generated-by: Claude Fable 5.1
---
.asf.yaml | 2 +-
.github/dependabot.yml | 4 +-
.github/workflows/runs-on-usage-report.yml | 4 +-
.github/workflows/terraform-runs-on-flex.yml | 4 +-
.github/workflows/terraform-runs-on.yml | 118 ------------
README.rst | 6 +-
terraform/runs-on-access/README.rst | 67 ++++---
terraform/runs-on-access/main.tf | 205 +--------------------
terraform/runs-on-access/runs-on-flex.tf | 16 +-
terraform/runs-on-flex/README.rst | 55 +++---
terraform/runs-on-flex/main.tf | 4 +-
terraform/runs-on/.terraform.lock.hcl | 28 ---
terraform/runs-on/README.rst | 259 ---------------------------
terraform/runs-on/main.tf | 32 ----
terraform/runs-on/provider.tf | 31 ----
terraform/runs-on/runs-on.yml.example | 91 ----------
terraform/runs-on/state.tf | 28 ---
terraform/runs-on/variables.tf | 65 -------
terraform/runs-on/version.tf | 27 ---
19 files changed, 78 insertions(+), 968 deletions(-)
diff --git a/.asf.yaml b/.asf.yaml
index 9f09211..207b1c4 100644
--- a/.asf.yaml
+++ b/.asf.yaml
@@ -62,7 +62,7 @@ github:
oidc_subject:
use_immutable_subject: true
environments:
- # Gates terraform apply of terraform/runs-on: the AWS apply role trusts
only this environment.
+ # Gates terraform apply of terraform/runs-on-flex: the AWS apply role
trusts only this environment.
runs-on-prod:
required_reviewers:
- id: potiuk
diff --git a/.github/dependabot.yml b/.github/dependabot.yml
index ebc5af7..e0bd437 100644
--- a/.github/dependabot.yml
+++ b/.github/dependabot.yml
@@ -52,11 +52,9 @@ updates:
patterns: ["*"]
# Updates the providers in .terraform.lock.hcl and the version constraints
in version.tf. A
- # terraform/runs-on change is planned on its pull request and applied after
merge, behind the
- # runs-on-prod approval.
+ # terraform/runs-on-access change is applied by hand by an account admin
after merge.
- package-ecosystem: terraform
directories:
- - /terraform/runs-on
- /terraform/runs-on-access
schedule:
interval: weekly
diff --git a/.github/workflows/runs-on-usage-report.yml
b/.github/workflows/runs-on-usage-report.yml
index c516598..5da7138 100644
--- a/.github/workflows/runs-on-usage-report.yml
+++ b/.github/workflows/runs-on-usage-report.yml
@@ -92,8 +92,8 @@ jobs:
gh issue create --title "${ISSUE_TITLE}" --body "${summary}
The RunsOn stack launches Spot first and falls back to on-demand
when Spot capacity is
- short or while its Spot circuit breaker is open
(\`SpotCircuitBreaker\` in
- \`terraform/runs-on/main.tf\`). On-demand instances cost more and
count against the
+ short or while its Spot circuit breaker is open
(\`spot_circuit_breaker\` in
+ \`terraform/runs-on-flex/main.tf\`). On-demand instances cost more
and count against the
smaller on-demand vCPU quota, so jobs queue once it is full. The
RunsOn CloudWatch
dashboard shows the circuit breaker state. This issue gets a comment
every week while the
share stays above ${THRESHOLD_PERCENT}%, and is closed once it is
back under."
diff --git a/.github/workflows/terraform-runs-on-flex.yml
b/.github/workflows/terraform-runs-on-flex.yml
index 9981f49..5f10903 100644
--- a/.github/workflows/terraform-runs-on-flex.yml
+++ b/.github/workflows/terraform-runs-on-flex.yml
@@ -16,8 +16,8 @@
# under the License.
---
# Plans terraform/runs-on-flex on pull requests and on main; applies the plan
made on main once a
-# required reviewer approves the runs-on-prod environment. Same flow as
terraform-runs-on.yml; the
-# AWS roles live in terraform/runs-on-access and are applied by hand.
+# required reviewer approves the runs-on-prod environment. The AWS roles live
in
+# terraform/runs-on-access and are applied by hand.
name: Terraform (RunsOn Flex)
on: # yamllint disable-line rule:truthy
diff --git a/.github/workflows/terraform-runs-on.yml
b/.github/workflows/terraform-runs-on.yml
deleted file mode 100644
index 89d9123..0000000
--- a/.github/workflows/terraform-runs-on.yml
+++ /dev/null
@@ -1,118 +0,0 @@
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied. See the License for the
-# specific language governing permissions and limitations
-# under the License.
----
-# Plans terraform/runs-on on pull requests and on main; applies the plan made
on main once a
-# required reviewer approves the runs-on-prod environment. The AWS roles live
in
-# terraform/runs-on-access and are applied by hand.
-name: Terraform (RunsOn)
-
-on: # yamllint disable-line rule:truthy
- pull_request:
- # README and example edits must not request an apply; -detailed-exitcode
cannot filter no-op
- # plans because every plan shows the masked LicenseKey diff (see
terraform/runs-on/README.rst).
- paths:
- - "terraform/runs-on/*.tf"
- - "terraform/runs-on/.terraform.lock.hcl"
- - ".github/workflows/terraform-runs-on.yml"
- # Pull requests only: a plan proves the plan role still admits pull
requests after an account
- # admin applied the changed trust policies by hand. On main the change
needs no apply.
- - "terraform/runs-on-access/*.tf"
- push:
- branches: [main]
- paths:
- - "terraform/runs-on/*.tf"
- - "terraform/runs-on/.terraform.lock.hcl"
- - ".github/workflows/terraform-runs-on.yml"
-
-permissions: {}
-
-concurrency:
- group: terraform-runs-on-${{ github.ref }}
- cancel-in-progress: ${{ github.event_name == 'pull_request' }}
-
-env:
- AWS_REGION: us-east-2
- TF_IN_AUTOMATION: "true"
- TF_INPUT: "false"
-
-jobs:
- plan:
- # Fork pull requests get no OIDC token, so they cannot plan.
- if: >-
- github.event_name == 'push' ||
- github.event.pull_request.head.repo.full_name == github.repository
- runs-on: ubuntu-slim
- permissions:
- contents: read
- id-token: write
- defaults:
- run:
- working-directory: terraform/runs-on
- steps:
- - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 #
v7.0.1
- with:
- persist-credentials: false
- - uses:
aws-actions/configure-aws-credentials@e1253824e5c10ff9df46874f81ed3ec929e19cfd
# v6.3.0
- with:
- role-to-assume: arn:aws:iam::827901512104:role/runs-on-terraform-plan
- aws-region: ${{ env.AWS_REGION }}
- - uses:
hashicorp/setup-terraform@dfe3c3f87815947d99a8997f908cb6525fc44e9e # v4.0.1
- with:
- terraform_version: 1.16.2
- terraform_wrapper: false
- - run: terraform init -lockfile=readonly
- # The plan role cannot write the lock file; apply takes the lock and
refuses a stale plan.
- - run: terraform plan -lock=false -out=runs-on.tfplan
- - if: github.event_name == 'push'
- uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9
# v7.0.2
- with:
- name: runs-on-tfplan
- path: terraform/runs-on/runs-on.tfplan
- retention-days: 7
-
- apply:
- needs: plan
- if: github.event_name == 'push'
- # Not ubuntu-slim: its 15-minute job limit is shorter than a stack update
may take (the stack's
- # timeout_in_minutes is 30), and a killed apply leaves the state lock held.
- runs-on: ubuntu-24.04
- # Required reviewers and the main-only branch policy come from .asf.yaml.
- environment: runs-on-prod
- permissions:
- contents: read
- id-token: write
- defaults:
- run:
- working-directory: terraform/runs-on
- steps:
- - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 #
v7.0.1
- with:
- persist-credentials: false
- - uses:
aws-actions/configure-aws-credentials@e1253824e5c10ff9df46874f81ed3ec929e19cfd
# v6.3.0
- with:
- role-to-assume:
arn:aws:iam::827901512104:role/runs-on-terraform-apply
- aws-region: ${{ env.AWS_REGION }}
- - uses:
hashicorp/setup-terraform@dfe3c3f87815947d99a8997f908cb6525fc44e9e # v4.0.1
- with:
- terraform_version: 1.16.2
- terraform_wrapper: false
- - run: terraform init -lockfile=readonly
- - uses:
actions/download-artifact@9000827ccba6bdab643e8b6fd33ac0654aef8333 # v8.0.2
- with:
- name: runs-on-tfplan
- path: terraform/runs-on
- - run: terraform apply runs-on.tfplan
diff --git a/README.rst b/README.rst
index 304beb5..e33ef28 100644
--- a/README.rst
+++ b/README.rst
@@ -23,7 +23,7 @@ RunsOn deployment
`AIP-118
<https://cwiki.apache.org/confluence/spaces/AIRFLOW/pages/451974652>`_ selected
`RunsOn <https://runs-on.com/>`_ for Airflow's self-hosted GitHub Actions
runners. The Terraform
-module that creates its CloudFormation stack, and the runner presets, live in
-`terraform/runs-on <terraform/runs-on/README.rst>`_; the ``Terraform
(RunsOn)`` workflow plans
-and applies it, with the AWS roles from
+module that deploys its stack lives in
+`terraform/runs-on-flex <terraform/runs-on-flex/README.rst>`_; the ``Terraform
(RunsOn Flex)``
+workflow plans and applies it, with the AWS roles from
`terraform/runs-on-access <terraform/runs-on-access/README.rst>`_.
diff --git a/terraform/runs-on-access/README.rst
b/terraform/runs-on-access/README.rst
index 8b48218..d450d5b 100644
--- a/terraform/runs-on-access/README.rst
+++ b/terraform/runs-on-access/README.rst
@@ -18,43 +18,42 @@
AWS access for the RunsOn Terraform workflow
============================================
-The IAM roles that let ``.github/workflows/terraform-runs-on.yml`` plan and
apply
-`terraform/runs-on <../runs-on/README.rst>`_ without any stored AWS
credential. This module is
-applied **by hand by an account admin** (own state key
``tf-state/runs-on-access.tfstate``) and
-never by the workflow, so the workflow cannot widen its own permissions.
+The IAM roles that let ``.github/workflows/terraform-runs-on-flex.yml`` plan
and apply
+`terraform/runs-on-flex <../runs-on-flex/README.rst>`_ without any stored AWS
credential, and the
+role of the weekly usage report. This module is applied **by hand by an
account admin** (own state
+key ``tf-state/runs-on-access.tfstate``) and never by the workflow, so the
workflow cannot widen
+its own permissions.
Roles
-----
-``runs-on-terraform-plan``
+``runs-on-terraform-flex-plan``
Assumed through GitHub OIDC by pull requests of
``apache/airflow-ci-infra`` and by pushes to
- ``main``. Reads the stack, the state and the license-key parameter
**without** decryption
- (``kms:Decrypt`` is denied explicitly: the ``aws/ssm`` key policy would
otherwise let
- ``ssm:GetParameter`` return the plain key). Plans run with
``-lock=false``; the role writes
- nothing. Fork pull requests get no OIDC token and cannot use it.
-``runs-on-terraform-apply``
+ ``main``. ``ReadOnlyAccess``, which refreshing the module's resources
needs, with denies that
+ keep secrets out of a pull request's reach: no S3 object but this module's
state, no
+ ``kms:Decrypt`` through SSM (the ``aws/ssm`` key policy would otherwise let
+ ``ssm:GetParameter`` return the plain license key) and no Secrets Manager
values. Plans run
+ with ``-lock=false``; the role writes nothing. Fork pull requests get no
OIDC token and cannot
+ use it.
+``runs-on-terraform-flex-apply``
Assumed only by jobs in the ``runs-on-prod`` environment, which
``.asf.yaml`` limits to
``main`` and to deployments approved by a required reviewer other than the
person who
- triggered them. It can call ``cloudformation:UpdateStack`` on the
``runs-on`` stack only with
- one of the release templates RunsOn publishes
-
(``https://runs-on.s3.eu-west-1.amazonaws.com/cloudformation/template-vX.Y.Z.yaml``,
matched
- with ``?`` globs so pre-release and other objects in the bucket do not
qualify), so a RunsOn
- upgrade or rollback needs only the reviewed ``runs_on_version`` change in
- ``terraform/runs-on``. It can also delete the stack, pass the service role
below and write
- the state and its lock file. It holds no infrastructure permissions.
-``runs-on-cloudformation``
- The stack's CloudFormation service role, which performs every create,
update and delete.
- RunsOn documents only "admin rights" for deploying its stack, and its
template touches IAM,
- VPC, ECS, Lambda, API Gateway, SQS, DynamoDB, S3, SNS and Budgets, with a
set that changes
- between releases, so the role has ``AdministratorAccess`` with explicit
denies on top: IAM
- writes only to the roles and instance profile the template names
- (``runs-on-<logical id>-*``) and to service-linked roles, no
``sts:AssumeRole*``, and no
- access to the ``airflow-ci-tfstate`` bucket. The role may attach only the
two managed
- policies the template uses, so the ``AppCustomPolicy`` /
``RunnerCustomPolicy`` stack
- parameters cannot give the stack's roles (or the CI runners) more
permissions; a custom policy
- needs that list extended here first. A template release that adds an IAM
resource
- with a new logical-ID prefix fails its update with ``AccessDenied`` until
the list in
- ``main.tf`` is extended.
+ triggered them. It creates the stack's resources itself. RunsOn documents
only "admin rights"
+ for deploying its stack, and the module's resource set changes between
releases, so the role
+ has ``AdministratorAccess`` with explicit denies on top
(``runs-on-flex.tf``): IAM writes only
+ to ``runs-on-flex-*`` roles and instance profiles and to service-linked
roles, every role it
+ creates must carry the ``runs-on-flex-boundary`` permissions boundary,
only the two managed
+ policies the module attaches, no ``sts:AssumeRole*``, no S3 but the
stack's cache bucket
+ (``runs-on-flex-cache-*``) and this module's state and lock file, and
us-east-2 only (IAM,
+ STS and Budgets are global). A module release that names an IAM role
outside that prefix or
+ attaches another managed policy fails its apply with ``AccessDenied``
until the lists in
+ ``runs-on-flex.tf`` are extended.
+``runs-on-flex-boundary``
+ The permissions boundary of every role the stack creates, so the ceiling
of what a Lambda
+ function, an ECS task or a runner instance of the stack can do whatever
its own policies say:
+ no IAM writes, ``iam:PassRole`` and ``sts:AssumeRole`` only on the stack's
own roles,
+ us-east-2 only (plus the global endpoints the module's roles call) and no
access to the
+ ``airflow-ci-tfstate`` bucket.
``runs-on-usage-report``
Assumed through GitHub OIDC by workflows running on ``main`` of
``apache/airflow-ci-infra``,
for the weekly ``.github/workflows/runs-on-usage-report.yml``. It can only
call
@@ -93,13 +92,11 @@ Bootstrap order
cd terraform/runs-on-access
terraform init
- terraform plan -out=runs-on-access.tfplan # 4 roles and their
policies, nothing else
+ terraform plan -out=runs-on-access.tfplan # 3 roles, their policies
and the boundary
terraform apply runs-on-access.tfplan
-5. The same admin applies ``terraform/runs-on`` once by hand, which sets
``iam_role_arn`` on the
- stack (the workflow would only do it on the next push that changes
``terraform/runs-on``'s
- ``.tf`` files).
- From then on the workflow plans and applies.
+ From then on the ``Terraform (RunsOn Flex)`` workflow plans and applies
+ ``terraform/runs-on-flex``.
Switching to the immutable subject
----------------------------------
diff --git a/terraform/runs-on-access/main.tf b/terraform/runs-on-access/main.tf
index fe3d3de..425369f 100644
--- a/terraform/runs-on-access/main.tf
+++ b/terraform/runs-on-access/main.tf
@@ -15,19 +15,10 @@
# specific language governing permissions and limitations
# under the License.
+# What every role here shares: the GitHub OIDC trust of
apache/airflow-ci-infra. The roles of the
+# Terraform (RunsOn Flex) workflow are in runs-on-flex.tf.
locals {
account_id = data.aws_caller_identity.current.account_id
- stack_arn =
"arn:aws:cloudformation:us-east-2:${local.account_id}:stack/runs-on/*"
- state = "arn:aws:s3:::airflow-ci-tfstate/tf-state/runs-on.tfstate"
-
- # Any release template RunsOn publishes (vX.Y.Z, one-digit major, one- or
two-digit minor and
- # patch), so an upgrade (or a rollback) is a reviewed change of
runs_on_version in
- # terraform/runs-on alone. "?" matches exactly one character, unlike "*", so
pre-release or other
- # objects in the bucket (template-v3.6.0-rc1.yaml, sub-paths) and other
buckets stay out of reach.
- runs_on_template_urls = [
- for p in setproduct(["?", "??"], ["?", "??"]) :
-
"https://runs-on.s3.eu-west-1.amazonaws.com/cloudformation/template-v?.${p[0]}.${p[1]}.yaml";
- ]
# The repository is pinned by its immutable IDs (repository_owner_id and
repository_id claims),
# so a renamed or re-created repository cannot match. Fork pull requests get
no OIDC token at all,
@@ -50,21 +41,6 @@ locals {
}
# The weekly report runs on a schedule, so only main.
usage_report_subjects = [for s in local.github_subject_prefixes :
"${s}:ref:refs/heads/main"]
-
- # IAM names the RunsOn template generates for the runs-on stack:
<stack>-<logical id>-<suffix>.
- # Matching the logical-ID prefixes rather than "runs-on-*" keeps every other
role in the account,
- # including the ones defined here, out of reach of the CloudFormation
service role.
- stack_role_prefixes = [
- "RunsOn", "EC2Instance", "ManagedLaunchTemplate", "SchedulerInvoke",
"StackConfigMaterializer",
- "NetworkFirewallEndpointResolver", "AlertTopicSlackWebhook",
- ]
- stack_iam_resources = concat(
- [for prefix in local.stack_role_prefixes :
"arn:aws:iam::${local.account_id}:role/runs-on-${prefix}*"],
- [
-
"arn:aws:iam::${local.account_id}:instance-profile/runs-on-EC2InstanceProfile*",
- "arn:aws:iam::${local.account_id}:role/aws-service-role/*",
- ],
- )
}
data "aws_caller_identity" "current" {}
@@ -106,110 +82,6 @@ data "aws_iam_policy_document" "github_trust" {
}
}
-resource "aws_iam_role" "terraform" {
- for_each = local.github_subjects
-
- name = "runs-on-terraform-${each.key}"
- description = "terraform ${each.key} of terraform/runs-on from
apache/airflow-ci-infra GitHub Actions"
- assume_role_policy = data.aws_iam_policy_document.github_trust[each.key].json
-}
-
-# What terraform plan needs: refresh the stack, prove the license-key
parameter exists and read
-# the state. Plans run with -lock=false, so this policy grants no writes.
-data "aws_iam_policy_document" "read" {
- statement {
- actions = ["cloudformation:DescribeStacks",
"cloudformation:DescribeStackEvents", "cloudformation:GetTemplate"]
- resources = [local.stack_arn]
- }
- statement {
- actions = ["ssm:GetParameter"]
- resources =
["arn:aws:ssm:us-east-2:${local.account_id}:parameter/runs-on/license-key"]
- }
- # The aws/ssm key policy lets any principal in the account decrypt through
SSM, so
- # GetParameter alone would also return the plain license key. Terraform
never asks for it. The
- # deny is limited to SSM so an SSE-KMS state bucket would stay readable.
- statement {
- effect = "Deny"
- actions = ["kms:Decrypt"]
- resources = ["*"]
- condition {
- test = "StringEquals"
- variable = "kms:ViaService"
- values = ["ssm.us-east-2.amazonaws.com"]
- }
- }
- statement {
- actions = ["s3:ListBucket"]
- resources = ["arn:aws:s3:::airflow-ci-tfstate"]
- condition {
- test = "StringEquals"
- variable = "s3:prefix"
- values = ["tf-state/runs-on.tfstate"]
- }
- }
- statement {
- actions = ["s3:GetObject"]
- resources = [local.state]
- }
-}
-
-resource "aws_iam_role_policy" "read" {
- for_each = aws_iam_role.terraform
-
- name = "terraform-read"
- role = each.value.name
- policy = data.aws_iam_policy_document.read.json
-}
-
-# The apply role can only resubmit the stack from RunsOn's published
templates, or delete it.
-# CloudFormation then acts as the stack's service role below, so this role
holds no
-# infrastructure permissions.
-data "aws_iam_policy_document" "apply" {
- statement {
- actions = ["cloudformation:UpdateStack"]
- resources = [local.stack_arn]
- condition {
- test = "StringLike"
- variable = "cloudformation:TemplateUrl"
- values = local.runs_on_template_urls
- }
- }
- # Own statement: the TemplateUrl condition above would deny a DeleteStack,
which carries no
- # template.
- statement {
- actions = ["cloudformation:DeleteStack"]
- resources = [local.stack_arn]
- }
- # The provider sends the default tags on every UpdateStack.
- statement {
- actions = ["cloudformation:TagResource", "cloudformation:UntagResource"]
- resources = [local.stack_arn]
- }
- statement {
- actions = ["iam:PassRole"]
- resources = [aws_iam_role.cloudformation.arn]
- condition {
- test = "StringEquals"
- variable = "iam:PassedToService"
- values = ["cloudformation.amazonaws.com"]
- }
- }
- statement {
- actions = ["s3:PutObject"]
- resources = [local.state]
- }
- statement {
- actions = ["s3:GetObject", "s3:PutObject", "s3:DeleteObject"]
- resources = ["${local.state}.tflock"]
- }
-}
-
-resource "aws_iam_role_policy" "apply" {
- name = "terraform-apply"
- role = aws_iam_role.terraform["apply"].name
- policy = data.aws_iam_policy_document.apply.json
-}
-
# Read-only role for .github/workflows/runs-on-usage-report.yml, which
compares the account's
# on-demand and Spot vCPU usage. GetMetricData has no resource-level
permissions.
resource "aws_iam_role" "usage_report" {
@@ -230,76 +102,3 @@ resource "aws_iam_role_policy" "usage_report" {
role = aws_iam_role.usage_report.name
policy = data.aws_iam_policy_document.usage_report.json
}
-
-data "aws_iam_policy_document" "cloudformation_trust" {
- statement {
- actions = ["sts:AssumeRole"]
- principals {
- type = "Service"
- identifiers = ["cloudformation.amazonaws.com"]
- }
- }
-}
-
-resource "aws_iam_role" "cloudformation" {
- name = "runs-on-cloudformation"
- description = "Service role CloudFormation uses to update the runs-on
stack"
- assume_role_policy = data.aws_iam_policy_document.cloudformation_trust.json
-}
-
-# RunsOn documents only "admin rights" for deploying its stack: the template
creates IAM, VPC,
-# ECS, Lambda, API Gateway, SQS, DynamoDB, S3, SNS and Budgets resources, and
the set changes
-# between releases. AdministratorAccess follows that; the denies below keep
the role away from
-# everything in the account that is not the stack's own.
-resource "aws_iam_role_policy_attachment" "cloudformation" {
- role = aws_iam_role.cloudformation.name
- policy_arn = "arn:aws:iam::aws:policy/AdministratorAccess"
-}
-
-data "aws_iam_policy_document" "cloudformation_guardrails" {
- statement {
- sid = "OnlyTheStackIam"
- effect = "Deny"
- actions = [
- "iam:Add*", "iam:Attach*", "iam:Change*", "iam:Create*",
"iam:Deactivate*", "iam:Delete*",
- "iam:Detach*", "iam:Enable*", "iam:PassRole", "iam:Put*", "iam:Remove*",
"iam:Reset*",
- "iam:Set*", "iam:Tag*", "iam:Untag*", "iam:Update*", "iam:Upload*",
- ]
- not_resources = local.stack_iam_resources
- }
- # Stack parameters (AppCustomPolicy, RunnerCustomPolicy) can attach any
managed policy to the
- # stack's own roles, which the IAM deny above exempts. The template itself
attaches only these
- # two.
- statement {
- sid = "OnlyTheTemplateManagedPolicies"
- effect = "Deny"
- actions = ["iam:AttachRolePolicy"]
- resources = ["*"]
- condition {
- test = "ArnNotEquals"
- variable = "iam:PolicyARN"
- values = [
- "arn:aws:iam::aws:policy/AmazonSSMManagedEC2InstanceDefaultPolicy",
-
"arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy",
- ]
- }
- }
- statement {
- sid = "NoRoleHopping"
- effect = "Deny"
- actions = ["sts:AssumeRole", "sts:AssumeRoleWithWebIdentity",
"sts:AssumeRoleWithSAML"]
- resources = ["*"]
- }
- statement {
- sid = "NoTerraformState"
- effect = "Deny"
- actions = ["s3:*"]
- resources = ["arn:aws:s3:::airflow-ci-tfstate",
"arn:aws:s3:::airflow-ci-tfstate/*"]
- }
-}
-
-resource "aws_iam_role_policy" "cloudformation_guardrails" {
- name = "guardrails"
- role = aws_iam_role.cloudformation.name
- policy = data.aws_iam_policy_document.cloudformation_guardrails.json
-}
diff --git a/terraform/runs-on-access/runs-on-flex.tf
b/terraform/runs-on-access/runs-on-flex.tf
index 06a98b6..ff4d5f9 100644
--- a/terraform/runs-on-access/runs-on-flex.tf
+++ b/terraform/runs-on-access/runs-on-flex.tf
@@ -15,13 +15,13 @@
# specific language governing permissions and limitations
# under the License.
-# Roles of the Terraform (RunsOn Flex) workflow, which applies
terraform/runs-on-flex: RunsOn's
-# Terraform module instead of its CloudFormation template. No CloudFormation
service role sits in
-# between, so the apply role itself creates the infrastructure. It gets what
the CloudFormation
-# service role in main.tf has (AdministratorAccess with denies) and three
limits more: it acts
-# only in us-east-2 (IAM and Budgets are global), it reaches no S3 bucket but
the stack's cache
-# buckets and its own state, and every role it creates carries the
runs-on-flex-boundary
-# permissions boundary, so no role of the stack can be given more than the
boundary allows.
+# Roles of the Terraform (RunsOn Flex) workflow, which applies
terraform/runs-on-flex from RunsOn's
+# Terraform module. No CloudFormation service role sits in between, so the
apply role itself
+# creates the infrastructure: AdministratorAccess with denies on top. It
writes IAM only under the
+# stack's own prefix, it acts only in us-east-2 (IAM and Budgets are global),
it reaches no S3
+# bucket but the stack's cache buckets and its own state, and every role it
creates carries the
+# runs-on-flex-boundary permissions boundary, so no role of the stack can be
given more than the
+# boundary allows.
locals {
flex_stack = "runs-on-flex"
flex_state = "arn:aws:s3:::airflow-ci-tfstate/tf-state/runs-on-flex.tfstate"
@@ -98,7 +98,7 @@ resource "aws_iam_role_policy" "flex_plan_guardrails" {
}
# RunsOn documents only "admin rights" for deploying the stack, and the
module's resource set
-# changes between releases, so AdministratorAccess as for the CloudFormation
service role.
+# changes between releases, so AdministratorAccess.
resource "aws_iam_role_policy_attachment" "flex_apply" {
role = aws_iam_role.flex_terraform["apply"].name
policy_arn = "arn:aws:iam::aws:policy/AdministratorAccess"
diff --git a/terraform/runs-on-flex/README.rst
b/terraform/runs-on-flex/README.rst
index 65a0646..157d113 100644
--- a/terraform/runs-on-flex/README.rst
+++ b/terraform/runs-on-flex/README.rst
@@ -18,17 +18,17 @@
RunsOn Flex — the runners from RunsOn's Terraform module
========================================================
-This module deploys the same RunsOn runners as `terraform/runs-on
<../runs-on/README.rst>`_, from
-RunsOn's `Terraform module
<https://github.com/runs-on/terraform-aws-runs-on>`_ (Flex, v3.5.1)
-instead of its CloudFormation template, into a VPC this module creates with a
public subnet in
-each of the three us-east-2 zones. Both stacks run side by side until the jobs
of
-``apache/airflow`` have moved here; the CloudFormation stack is then removed
in a separate change.
+This module deploys Airflow's RunsOn runners from RunsOn's
+`Terraform module <https://github.com/runs-on/terraform-aws-runs-on>`_ (Flex,
v3.5.1) into a VPC
+this module creates with a public subnet in each of the three us-east-2 zones.
It replaced the
+``runs-on`` CloudFormation stack (RunsOn's template, ``terraform/runs-on``
until its retirement),
+which served the jobs of ``apache/airflow`` until 2026-11-01.
Why this module
---------------
-The CloudFormation template places runners in the first two zones of the
region, use2-az1 and
-use2-az2 for this account, and has no parameter to change that. On 2026-10-30
those two zones
+The CloudFormation template placed runners in the first two zones of the
region, use2-az1 and
+use2-az2 for this account, and had no parameter to change that. On 2026-10-30
those two zones
had the worst Spot capacity in the region for the runners' instance families:
================================= ========== ========== ==========
@@ -48,10 +48,10 @@ Files
``network.tf``
VPC (``vpc_cidr``, default ``10.3.0.0/16``), internet gateway, one public
subnet per zone ID
- and an S3 gateway endpoint. No NAT: runners get public IPs, as on the
CloudFormation stack.
+ and an S3 gateway endpoint. No NAT: runners get public IPs.
``main.tf``
- The ``runs-on/runs-on/aws//modules/flex`` module with the CloudFormation
stack's settings, the
- license-key lookup and the permissions boundary of the stack's roles.
+ The ``runs-on/runs-on/aws//modules/flex`` module with the settings carried
over from the
+ CloudFormation stack, the license-key lookup and the permissions boundary
of the stack's roles.
``variables.tf``
All variables have defaults: ``stack_name`` (``runs-on-flex``),
``runs_on_environment``
(``flex``), ``github_organization``, ``alert_email``,
``enable_admin_routes``
@@ -59,13 +59,14 @@ Files
``state.tf``
S3 backend ``airflow-ci-tfstate`` / ``tf-state/runs-on-flex.tfstate``.
-How it differs from the CloudFormation stack
---------------------------------------------
+How it differs from the retired CloudFormation stack
+----------------------------------------------------
-- **Environment.** The stack takes only jobs whose label carries ``env=flex``;
the CloudFormation
- stack keeps the default ``production`` and ignores them. That is what lets
both run at once.
-- **Who creates the resources.** The CloudFormation stack is applied by a role
that can only
- resubmit RunsOn's published template; CloudFormation's service role does the
work. Here the
+- **Environment.** The stack takes only jobs whose label carries ``env=flex``.
A job without
+ ``env`` targets RunsOn's default ``production`` environment, which no stack
serves any more, so
+ it stays queued.
+- **Who creates the resources.** The CloudFormation stack was applied by a
role that could only
+ resubmit RunsOn's published template; CloudFormation's service role did the
work. Here the
``runs-on-terraform-flex-apply`` role creates everything itself, with
AdministratorAccess and
the denies in `terraform/runs-on-access/runs-on-flex.tf
<../runs-on-access/runs-on-flex.tf>`_:
IAM writes only on ``runs-on-flex-*`` roles and only with the
``runs-on-flex-boundary``
@@ -83,9 +84,8 @@ Setup
1. **Access (account admin, once).** Apply `terraform/runs-on-access
<../runs-on-access/README.rst>`_:
it adds the ``runs-on-terraform-flex-plan`` and ``-apply`` roles and the
``runs-on-flex-boundary`` policy.
-2. **License key (manual, once).** The same RunsOn license as the
CloudFormation stack, under
- this stack's name. The module receives only the parameter's ARN, so the key
never reaches
- Terraform state:
+2. **License key (manual, once).** The RunsOn license, under this stack's
name. The module
+ receives only the parameter's ARN, so the key never reaches Terraform state:
.. code-block:: bash
@@ -99,17 +99,12 @@ Setup
``Terraform (RunsOn Flex)`` workflow plans on ``main`` and applies once a
reviewer approves
``runs-on-prod``. Set ``enable_admin_routes = true`` in that first change.
4. **GitHub App (with ASF Infra).** Open the ``entry_point`` output →
**Register app**. This
- creates a second private GitHub App for this stack; ASF Infra installs it on
- ``apache/airflow`` (and ``.github-private``, for the presets). Then set
- ``enable_admin_routes`` back to ``false`` in a follow-up change.
-5. **Move the jobs.** In ``apache/airflow``, add ``/env=flex`` to the RunsOn
label templates in
- ``dev/breeze/src/airflow_breeze/global_constants.py``. All jobs of a run
must move together:
- the stash actions keep image tars in the cache bucket of the stack that ran
the job, so a
- job on the other stack would not find them. Rolling back is removing
``/env=flex``.
-6. **Retire the CloudFormation stack** once the runs have been green on this
stack for a while:
- a separate change in ``terraform/runs-on`` (remove ``prevent_destroy``,
destroy), the old
- GitHub App uninstalled by ASF Infra, and the ``runs-on-terraform-plan``,
``-apply`` and
- ``runs-on-cloudformation`` roles removed from ``runs-on-access``.
+ creates a private GitHub App for this stack; ASF Infra installs it on
``apache/airflow``. Then
+ set ``enable_admin_routes`` back to ``false`` in a follow-up change.
+5. **Point the jobs here.** In ``apache/airflow``, the RunsOn label templates
in
+ ``dev/breeze/src/airflow_breeze/global_constants.py`` carry ``/env=flex``.
All jobs of a run
+ must target the same stack: the stash actions keep image tars in the cache
bucket of the stack
+ that ran the job, so a job on another stack would not find them.
Locally, an account admin runs the same steps as the workflow::
diff --git a/terraform/runs-on-flex/main.tf b/terraform/runs-on-flex/main.tf
index 7ed8ce4..aefd3c5 100644
--- a/terraform/runs-on-flex/main.tf
+++ b/terraform/runs-on-flex/main.tf
@@ -42,7 +42,7 @@ module "runs_on" {
# Created in terraform/runs-on-access; the apply role cannot create a stack
role without it.
permission_boundary_arn =
"arn:aws:iam::${data.aws_caller_identity.current.account_id}:policy/${var.stack_name}-boundary"
- # The values of the CloudFormation stack (terraform/runs-on/main.tf);
everything else keeps the
+ # The values carried over from the retired CloudFormation stack; everything
else keeps the
# module default.
ssh_allowed = false
ssm_allowed = true
@@ -52,7 +52,7 @@ module "runs_on" {
cache_expiration_days = 10
app_size = "small"
# With hundreds of runners two interruptions in 15 minutes are routine, so
the default 2/15/30
- # keeps Spot off (terraform/runs-on/main.tf).
+ # keeps Spot off.
spot_circuit_breaker = "30/15/10"
enable_cost_reports = "daily"
app_budget_daily_usd = 10 # alert only, not a spending cap
diff --git a/terraform/runs-on/.terraform.lock.hcl
b/terraform/runs-on/.terraform.lock.hcl
deleted file mode 100644
index 5b3d3ef..0000000
--- a/terraform/runs-on/.terraform.lock.hcl
+++ /dev/null
@@ -1,28 +0,0 @@
-# This file is maintained automatically by "terraform init".
-# Manual edits may be lost in future updates.
-
-provider "registry.terraform.io/hashicorp/aws" {
- version = "6.70.0"
- constraints = "~> 6.0"
- hashes = [
- "h1:FTJvoIDVz+x6B26k+fxXHn0NOKMvVOP9qLz6VdycFkE=",
- "h1:HCr4MYgxlkFBOAeWKOoktNNzZaQX8ZZY4YTxmL/1V/Y=",
- "h1:L+05gOXA+kdYTtayIw0WxdaTp3JryZCMZpfrJVhoJA0=",
- "zh:068ca7ed4fc77e3d63d7fac54515a426a32f3dc143b55a6ad00963fe68289955",
- "zh:2d8c016b3809ab32f536b01bf2c98fcbdeb066b7b7e50f756ff6ff1e0b0539c7",
- "zh:450d960b60ea81268bd355600340787254f750d76e0075208a6fb903d7336b09",
- "zh:50447833cce21288ffc331342f9d83854e80515e2d86e8f5776a07ce18866b20",
- "zh:58acc041b080778ded75320e241ebcf6cf2ed145bede644b86a4784ce46086c6",
- "zh:7e899a44a1da95fefb1b04b231cf7cdb5871f482f60a710ede78bf25ac265724",
- "zh:8b141d13f4b84d7a47bc5a4855d0fc985191317b3249603532d4de03db5741c5",
- "zh:9035357f7d065ce8d35a9b05337977910683219c0f13fa09155d28f9b32e23a7",
- "zh:9b12af85486a96aedd8d7984b0ff811a4b42e3d88dad1a3fb4c0b580d04fa425",
- "zh:9ba0990a876d2641a185ca88263719afc4d5ac43533bdb147437f0073d2cd74d",
- "zh:a8773c7a02e1b3290df89351ee493fd8b74f807f92a7f69534e05901d4515761",
- "zh:ab0ebb298578eeeffe0fbf0a8b27ce99acd4419b7049272f1ab87e8c47229407",
- "zh:b88a3f22127bfb17604adc53e9bdb8ed0eddf4b9b77bf19871cba2e7fcd5b4a3",
- "zh:c7b1fb41e7f086b670f78932ede2b539903a8d71bac06f55f14a1a33d09a1ccf",
- "zh:df482bd93f51125f78feab7742aae1d7f22c7108ee2dc47787c4a1d611e6ccfd",
- "zh:fca401ffaa42d5f350d667123569de51a23de8c30b87ce0bd9d86fc80835fd4d",
- ]
-}
diff --git a/terraform/runs-on/README.rst b/terraform/runs-on/README.rst
deleted file mode 100644
index 33dbe16..0000000
--- a/terraform/runs-on/README.rst
+++ /dev/null
@@ -1,259 +0,0 @@
-.. Licensed to the Apache Software Foundation (ASF) under one
- or more contributor license agreements. See the NOTICE file
- distributed with this work for additional information
- regarding copyright ownership. The ASF licenses this file
- to you under the Apache License, Version 2.0 (the
- "License"); you may not use this file except in compliance
- with the License. You may obtain a copy of the License at
-
-.. http://www.apache.org/licenses/LICENSE-2.0
-
-.. Unless required by applicable law or agreed to in writing,
- software distributed under the License is distributed on an
- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- KIND, either express or implied. See the License for the
- specific language governing permissions and limitations
- under the License.
-
-RunsOn — self-hosted CI runners on AWS
-======================================
-
-`RunsOn <https://runs-on.com/>`_ is the approach selected by
-`AIP-118
<https://cwiki.apache.org/confluence/spaces/AIRFLOW/pages/451974652>`_ for
Airflow's
-self-hosted GitHub Actions runners. It packages runner provisioning into a
CloudFormation stack in
-Airflow's AWS account: a small control plane (ECS Fargate + API Gateway)
receives GitHub's
-``workflow_job`` webhooks and launches one ephemeral EC2 instance per job,
Spot first, sized from
-the job's ``runs-on:`` label. RunsOn's `free open-source license
<https://runs-on.com/billing/free/>`_
-covers the product; EC2/EBS/S3 usage is billed to the account.
-
-This module creates the stack for the **apache** organization from RunsOn's
published,
-version-pinned template, with the configuration validated on 2026-09-25 (full
``Tests (AMD)``
-graph green apart from two non-hermetic tests — see the AIP-118 appendix).
-
-Files
------
-
-``main.tf``
- The ``aws_cloudformation_stack`` resource, its parameters and the
license-key lookup.
-``variables.tf``
- All variables have defaults: ``stack_name`` (``runs-on``),
``github_organization``
- (``apache``), ``alert_email`` (``[email protected]``),
``runs_on_version``
- (``v3.5.1``), ``enable_admin_routes`` (``false``) and ``vpc_cidr``
(``10.2.0.0/16``). The
- region (``us-east-2``) is not a variable: it is hard-coded in
``provider.tf``, the backend in
- ``state.tf`` and the commands below, so a move changes all of them.
-``state.tf``
- S3 backend ``airflow-ci-tfstate`` / ``tf-state/runs-on.tfstate`` (its own
key next to the
- EKS states).
-``runs-on.yml.example``
- Runner presets for Airflow's three size classes, to be committed as
``.github/runs-on.yml``
- in the organization's ``.github-private`` repository (or a repository's
default branch).
-
-Step 1 — create the license-key parameter (manual, once)
---------------------------------------------------------
-
-RunsOn accepts ``LicenseKey`` either as plain text or as the ARN of an SSM
Parameter Store
-parameter that must be named ``/<stack name>/license-key`` (`reference
-<https://runs-on.com/docs/maintenance/configuration/flex/cloudformation/>`_).
The ARN form keeps
-the key out of Terraform variables, plan output and state: ``main.tf`` reads
the parameter
-**without decryption** (only to prove it exists) and passes its ARN; the
stack's service role is
-the only principal that decrypts it. Secrets Manager is not supported by the
template.
-
-.. code-block:: bash
-
- aws ssm put-parameter --region us-east-2 \
- --name /runs-on/license-key \
- --type SecureString \
- --description "RunsOn license key for the runs-on stack (GitHub
organization apache)" \
- --value 'RUNS-ON-LICENSE-KEY'
-
-Use the default SSM KMS key (``alias/aws/ssm``): the template grants the
stack's role access to
-that key; a customer key would need an extra key policy. Never put the key in
``terraform.tfvars``.
-Rotating the key does not go through Terraform: ``LicenseKey`` stays the same
ARN, so an apply
-does not make the stack read the new value. The control-plane service reads
the parameter itself,
-so overwrite it and restart the service:
-
-.. code-block:: bash
-
- aws ssm put-parameter --region us-east-2 --name /runs-on/license-key \
- --type SecureString --overwrite --value 'NEW-RUNS-ON-LICENSE-KEY'
- service="$(aws cloudformation describe-stack-resource --region us-east-2 \
- --stack-name runs-on --logical-resource-id RunsOnWorkerService \
- --query StackResourceDetail.PhysicalResourceId --output text)"
- aws ecs update-service --region us-east-2 --force-new-deployment \
- --cluster "$(cut -d / -f 2 <<< "${service}")" --service "${service}"
-
-The free open-source license is issued per GitHub organization; the key must
be the one issued
-for ``apache``.
-
-Step 2 — plan and apply
------------------------
-
-Changes go through pull requests. The ``Terraform (RunsOn)`` workflow
-(``.github/workflows/terraform-runs-on.yml``) runs only when this module's
``.tf`` files, its
-``.terraform.lock.hcl`` or the workflow file change; edits to this README or to
-``runs-on.yml.example`` do not trigger it. It plans such pull requests, plans
again on ``main``
-after the merge, and applies that plan once a required reviewer approves the
``runs-on-prod``
-environment. The plan artifact is kept 7 days; if approval comes later, use
"Re-run all jobs" on
-the run, which plans again. Do that only while the run is still the latest one
on ``main``:
-re-running replays the original commit, so once a newer run has applied,
re-running an older one
-plans a revert. The workflow authenticates with GitHub OIDC; the AWS roles it
assumes, and the
-CloudFormation service role the stack runs as, are in
-`terraform/runs-on-access <../runs-on-access/README.rst>`_, which is applied
by hand first.
-
-The state is in the S3 backend of ``state.tf``, locked with ``use_lockfile``.
An account admin can
-still run the same steps locally::
-
- cd terraform/runs-on
- terraform init
- terraform plan -out=runs-on.tfplan
- terraform apply runs-on.tfplan # ~10 minutes on creation
- terraform output
-
-The ``apache`` stack already exists: it was created on 2026-09-25 with local
state, which was then
-moved to S3 with ``terraform init -migrate-state``. A plan against it must
therefore never propose
-to create the stack; if it does, Terraform is not reading the S3 state, and an
apply would fail
-with ``AlreadyExistsException``. On a fresh account the plan shows exactly one
resource to create
-(``aws_cloudformation_stack.runs_on``) and reads two data sources (the SSM
parameter and the
-caller identity).
-
-Step 3 — register the GitHub App and the presets
-------------------------------------------------
-
-1. Apply with ``-var enable_admin_routes=true``, open the ``entry_point``
output → **Register
- app**, then apply again without the variable to remove the ``/setup``
routes from the public
- entry point. This creates a private GitHub App in the organization and
installs it; for
- ``apache`` this is done together with ASF Infra, who also grant the app
access to
- ``apache/airflow`` and to ``.github-private`` if the presets live there.
Re-registering the
- app needs the same temporary apply.
-2. Commit ``runs-on.yml.example`` as ``.github/runs-on.yml`` in
``apache/.github-private``
- (stack parameter ``RunnerConfigAutoExtendsFrom``, default
``.github-private``). RunsOn reads
- ``.github/runs-on.yml`` in a public repository **only from its default
branch**, so presets
- cannot be trialled from a feature branch; the validation therefore spelled
the same values
- out in the labels.
-3. Point jobs at the runners:
-
- .. code-block:: yaml
-
- runs-on: runs-on=${{ github.run_id }}/runner=airflow-amd-medium
-
- ``runs-on=<run id>`` is mandatory (it routes the job and namespaces the
runner). Labels can
- also carry everything inline, which is what the validation did:
-
``runs-on=<id>/family=m7a+m7i+m6a+m6i/image=ubuntu22-full-x64/volume=150gb/cpu=8/ram=32``.
- ``runs-on/action`` (per-job cost line in the job summary) is already on the
ASF GitHub Actions
- allowlist.
-
-Upgrades and changes
---------------------
-
-Bump ``runs_on_version`` (or any parameter) and ``terraform plan`` /
``apply``: the resource
-always resubmits the template URL, which is what RunsOn's `upgrade guide
-<https://runs-on.com/docs/maintenance/upgrades/>`_ requires (a reused
processed template can
-silently skip parameter changes). Minor and patch releases have no downtime;
major releases may.
-Rollback is the previous ``runs_on_version``. The apply role may deploy any
release template from
-RunsOn's bucket (``template-vX.Y.Z.yaml``,
-`terraform/runs-on-access <../runs-on-access/README.rst>`_) and
``runs_on_version`` must look like
-``v3.5.1``, so an upgrade or rollback needs no admin step.
-What still needs an admin is IAM: the CloudFormation service role may only
manage the IAM roles
-named in ``stack_role_prefixes`` and attach the two managed policies listed in
-``terraform/runs-on-access``. Before a bump compare the new template's
``AWS::IAM::Role``,
-``AWS::IAM::InstanceProfile`` and ``ManagedPolicyArns`` with those lists;
anything new fails the
-stack update until an admin adds it there and applies that module.
-Template SHA-256s: ``v3.3.2``
``3017805d77c1d717fdd229a1e1242db12c985944699d3d8dca01f6f28b35c16d``,
-``v3.5.1``
``9acd6fd154e2bb921f95466c45380d2e3d7bd718553a12af4e5c76d15915bd57``.
-
-Every plan shows ``"LicenseKey" = "****" ->
"arn:...:parameter/runs-on/license-key"``:
-CloudFormation masks ``NoEcho`` parameters when Terraform reads the stack, and
the AWS provider
-stores the mask. The value sent on apply is the same ARN, so that line alone
changes nothing. Do
-not add it to ``ignore_changes``: the provider would then send the literal
``****`` as the license
-key.
-
-``prevent_destroy`` is set on the stack: a ``terraform destroy`` fails until
the lifecycle block is
-removed on purpose. Deleting the stack also needs its cache bucket emptied
first if CloudFormation
-stops on it.
-
-What the stack contains
------------------------
-
-VPC ``10.2.0.0/16`` with public and private subnets (``Private=false`` →
runners get public IPs and
-no NAT, so each runner pulls from Docker Hub from its own address), an ECS
Fargate service for the
-control plane, an API Gateway entry point (``/github/webhooks``, plus ``/``,
``/setup`` and
-``/readyz`` only while ``enable_admin_routes`` is true), an S3 cache bucket
(``cache_bucket``
-output; the runners' instance role can read and write the ``cache/`` prefix,
lifecycle
-``S3CacheExpirationInDays``), the EC2 instance role and launch templates,
DynamoDB tables for
-locks and job tracking, an SNS alert topic, an AWS Budget
(``AppBudgetDailyUsd``,
-alert only — not a spending cap), the CloudWatch dashboard
``RunsOnDashboard-*`` (queue health,
-scheduling trend, queue-duration P50/P90, Spot circuit breaker), log groups
-``/aws/ecs/<stack>/runs-on-worker`` and ``<stack>-EC2InstanceLogGroup-*``, and
a daily cost report
-by email.
-
-Observability
--------------
-
-- CloudWatch dashboard and log groups above; ``roc`` CLI (`docs
- <https://runs-on.com/docs/observability/cli/>`_): ``roc stack doctor``,
``roc stack logs``,
- ``roc logs <job url> --watch``, ``roc connect <job url>`` over SSM
(``SSMAllowed=true``).
-- ``.github/workflows/runs-on-usage-report.yml`` checks every Monday how much
of the last week's
- EC2 vCPU-hours in us-east-2 ran on on-demand instances. Above 30% it keeps
one issue open in
- this repository and comments each week's numbers on it, and it closes the
issue once the share
- is back under. The numbers are account-wide (``AWS/Usage`` metrics), so any
other EC2
- instances in the region count too.
-- Optional OTLP export (``OtelExporterEndpoint`` / ``OtelExporterHeaders`` +
``extras=otel`` on the
- label) to SigNoz or any OTLP backend; RunsOn publishes a SigNoz dashboard.
Not deployed by
- the stack.
-
-Things learned in the validation
---------------------------------
-
-- Runner instances use IMDSv2 with a hop limit of 2, so Docker containers
started by a job can
- obtain the instance role's credentials. Tests that assume "no AWS
credentials" must set
- ``AWS_EC2_METADATA_DISABLED=true`` (fixed in ``apache/airflow#73707``). The
same credentials
- can read and write the whole ``cache/`` prefix, so any job on these runners
can overwrite
- objects that later runs download.
-- Untrusted code is meant to stay off these runners. Airflow's CI
(``apache/airflow#74889``)
- routes only pushes, schedules and dispatches on ``apache/airflow``, branches
of
- ``apache/airflow`` itself, and pull requests labelled ``use self-hosted
runners`` that a
- committer approved at their head commit to RunsOn; everything else runs on
GitHub-hosted
- runners. That routing is cooperative, not a boundary: a ``pull_request`` run
uses the workflow
- file of the pull request, so a fork can ask for a RunsOn label itself. The
boundary is GitHub's
- fork-workflow approval, which must be set to "Require approval for all
external contributors"
- for ``apache/airflow`` (ASF Infra).
-- Airflow's stash actions (image tars, mount caches, prek cache) store
payloads in the stack's
- cache bucket under ``cache/airflow-stash/``, so multi-gigabyte payloads
never leave the region.
- Payloads are content-addressed
(``cache/airflow-stash/sha256/<digest>.tar``); the digest itself
- is kept as a GitHub-scoped ASF stash, and a restore whose download does not
match it is a cache
- miss. Overwriting an object in the bucket therefore cannot change what a
later run restores.
-- Job pickup (job queued → runner started) was 29 s at the median, 51 s at
p90; a full
- ``Tests (AMD)`` graph finished in 53 minutes.
-- Magic cache (``extras=s3-cache``) was not enabled: it only speeds up
``actions/cache`` and
- ``setup-*`` caches and requires ``runs-on/action`` before every
``upload-artifact``.
-- EC2 quotas in us-east-2 (``All Standard Spot Instance Requests`` 3000 vCPU,
on-demand 2000)
- were enough for ~130 concurrent jobs.
-
-Spot capacity
--------------
-
-On 2026-10-25..27 the stack launched 4706 Spot and 4038 on-demand runners. The
worker log
-(``force_on_demand_reason`` on each "Using on-demand instances" line) gives
the cause, and none
-of it was the Spot quota (peak 2332 of 3000 vCPU); the on-demand quota filled
up instead
-(2004 of 2000 on 2026-10-26 16:00 UTC, 1653 ``VcpuLimitExceeded``):
-
-- ``spot_disabled`` (2641): the default ``SpotCircuitBreaker`` ``2/15/30``
kept Spot off; none
- since the ``30/15/10`` apply on 2026-10-26 20:29 UTC.
-- ``force_on_demand`` (678): x64 Spot capacity misses. Up to v3.4 one miss
sends every job to
- on-demand for five minutes; from v3.5.0 only the affected runners go.
-- ``spot_recovery_rerun`` (719): RunsOn's rerun of an interrupted job,
on-demand by design.
-
-Of 2156 Spot capacity errors in a day, 1959 were x64 (``m7a+m7i+m6a+m6i``) and
197 arm64.
-More instance families give the Spot allocator more pools, so the x64 presets
in
-``runs-on.yml.example`` also allow ``m8a``, ``m8i``, ``m5a`` and ``m5`` (Spot
``2xlarge`` on
-2026-10-27: 0.08-0.16 USD/hour against 0.12-0.15 for the existing families).
Since
-``apache/airflow#75822`` the jobs (which spell their labels out instead of
using these presets)
-also take Spot from capacity-optimized pools (``spot=co``) and accept up to
twice their size
-(``cpu=8+16/ram=32+64``). Capacity-optimized ignores price, so a job can land
on the larger size
-at up to twice the hourly price; the weekly report and the daily cost report
show whether the net
-cost goes down.
-
-The CloudFormation template places runners in two Availability Zones
(us-east-2a and 2b,
-fixed by the template). RunsOn's Spot spread across zones
(``RUNS_ON_SPOT_FORCE_ZONE_SPREAD``,
-v3.5.0 beta) and custom subnets exist only in its Terraform module, so a third
zone means
-moving off the CloudFormation stack.
diff --git a/terraform/runs-on/main.tf b/terraform/runs-on/main.tf
deleted file mode 100644
index f76f3af..0000000
--- a/terraform/runs-on/main.tf
+++ /dev/null
@@ -1,32 +0,0 @@
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied. See the License for the
-# specific language governing permissions and limitations
-# under the License.
-
-# The license key is never in Terraform state: RunsOn accepts LicenseKey as
the ARN of an SSM
-# SecureString parameter that MUST be named /<stack name>/license-key, and
only the stack's
-# service role decrypts it. The parameter is created by hand (see README.rst);
this lookup
-# without decryption only proves it exists before the stack is planned,
storing the ciphertext.
-data "aws_ssm_parameter" "license_key" {
- name = "/${var.stack_name}/license-key"
- with_decryption = false
-}
-
-data "aws_caller_identity" "current" {}
-
-# The stack was removed from this configuration on purpose: every job of
apache/airflow runs on
-# terraform/runs-on-flex since 2026-11-01, and applying this change is what
deletes the stack
-# (the Terraform (RunsOn) workflow, after the runs-on-prod approval). The
template retains the
-# cache and access-log buckets (DeletionPolicy RetainExceptOnCreate); delete
them by hand.
diff --git a/terraform/runs-on/provider.tf b/terraform/runs-on/provider.tf
deleted file mode 100644
index 452a66a..0000000
--- a/terraform/runs-on/provider.tf
+++ /dev/null
@@ -1,31 +0,0 @@
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied. See the License for the
-# specific language governing permissions and limitations
-# under the License.
-
-# Credentials come from the environment (AWS_PROFILE locally, OIDC role in the
-# future GitHub Actions workflow) — do not hardcode a profile here.
-provider "aws" {
- # Also hard-coded in the backend (state.tf) and the README commands; change
all together.
- region = "us-east-2"
-
- default_tags {
- tags = {
- Project = "airflow-ci"
- ManagedBy = "terraform"
- Repo = "apache/airflow-ci-infra"
- }
- }
-}
diff --git a/terraform/runs-on/runs-on.yml.example
b/terraform/runs-on/runs-on.yml.example
deleted file mode 100644
index ee45cbd..0000000
--- a/terraform/runs-on/runs-on.yml.example
+++ /dev/null
@@ -1,91 +0,0 @@
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied. See the License for the
-# specific language governing permissions and limitations
-# under the License.
----
-# RunsOn runner presets for the Airflow CI size classes, as
`.github/runs-on.yml`.
-#
-# RunsOn reads this file only from a public repository's default branch, or
from the
-# organisation's `.github-private` repository (stack parameter
RunnerConfigAutoExtendsFrom).
-# The 2026-09-25 validation could not use presets for that reason and spelled
the same values
-# out in job labels instead; with presets in place a job label becomes
-# `runs-on=${{ github.run_id }}/runner=airflow-amd-medium`.
-#
-# Classes are the EC2 equivalents of the measured EKS classes (AIP-118
appendix):
-# small 4 vCPU / 16 GB, medium 8 / 32, xl 32 / 128. The xl class gets 128 GB
because xdist
-# parallelism follows the host's vCPUs. Each class also accepts up to twice
its size, and Spot comes
-# from capacity-optimized pools (`spot: co`), as in the job labels of
apache/airflow#75822; RunsOn
-# reruns an interrupted job on on-demand. `volume` replaces the GitHub-hosted
runners' root + /mnt
-# disks.
-runners:
- airflow-amd-small:
- cpu: [4, 8]
- ram: [16, 32]
- family: ["m8a", "m8i", "m7a", "m7i", "m6a", "m6i", "m5a", "m5"]
- image: ubuntu22-full-x64
- volume: 150gb
- spot: co
- airflow-amd-medium:
- cpu: [8, 16]
- ram: [32, 64]
- family: ["m8a", "m8i", "m7a", "m7i", "m6a", "m6i", "m5a", "m5"]
- image: ubuntu22-full-x64
- volume: 150gb
- spot: co
- airflow-amd-xl:
- cpu: [32, 64]
- ram: [128, 256]
- family: ["m8a", "m8i", "m7a", "m7i", "m6a", "m6i", "m5a", "m5"]
- image: ubuntu22-full-x64
- volume: 150gb
- spot: co
- airflow-arm-small:
- cpu: [4, 8]
- ram: [16, 32]
- family: ["m8g", "m7g", "m6g"]
- image: ubuntu22-full-arm64
- volume: 150gb
- spot: co
- airflow-arm-medium:
- cpu: [8, 16]
- ram: [32, 64]
- family: ["m8g", "m7g", "m6g"]
- image: ubuntu22-full-arm64
- volume: 150gb
- spot: co
- airflow-arm-xl:
- cpu: [32, 64]
- ram: [128, 256]
- family: ["m8g", "m7g", "m6g"]
- image: ubuntu22-full-arm64
- volume: 150gb
- spot: co
- airflow-amd-in-memory:
- cpu: [8, 16]
- ram: [32, 128]
- family: ["m8a", "m8i", "m7a", "m7i", "m6a", "m6i", "m5a", "m5",
- "r8i", "r7a", "r7i", "r6a", "r6i"]
- image: ubuntu22-full-x64
- volume: 150gb
- spot: co
- extras: ["tmpfs"]
- airflow-arm-in-memory:
- cpu: [8, 16]
- ram: [32, 128]
- family: ["m8g", "m7g", "m6g", "r8g", "r7g", "r6g"]
- image: ubuntu22-full-arm64
- volume: 150gb
- spot: co
- extras: ["tmpfs"]
diff --git a/terraform/runs-on/state.tf b/terraform/runs-on/state.tf
deleted file mode 100644
index 36b2b44..0000000
--- a/terraform/runs-on/state.tf
+++ /dev/null
@@ -1,28 +0,0 @@
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied. See the License for the
-# specific language governing permissions and limitations
-# under the License.
-
-terraform {
- backend "s3" {
- region = "us-east-2"
- bucket = "airflow-ci-tfstate"
- # Own key: the EKS deployment (tf-state/eks-airflow-ci.tfstate) and the
2024
- # state (tf-state/eks.tfstate) must not be mixed with this one.
- key = "tf-state/runs-on.tfstate"
- encrypt = true
- use_lockfile = true
- }
-}
diff --git a/terraform/runs-on/variables.tf b/terraform/runs-on/variables.tf
deleted file mode 100644
index 7998cb1..0000000
--- a/terraform/runs-on/variables.tf
+++ /dev/null
@@ -1,65 +0,0 @@
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied. See the License for the
-# specific language governing permissions and limitations
-# under the License.
-
-variable "stack_name" {
- description = <<-EOT
- CloudFormation stack name. RunsOn derives the license-key SSM parameter
name from it
- (/<stack_name>/license-key), so changing it means re-creating the
parameter.
- EOT
- type = string
- default = "runs-on"
-}
-
-variable "runs_on_version" {
- description = "RunsOn release to deploy; selects the version-pinned
CloudFormation template."
- type = string
- default = "v3.5.1"
-
- # The apply role may deploy only release templates of this shape from
RunsOn's bucket
- # (runs_on_template_urls in terraform/runs-on-access).
- validation {
- condition = can(regex("^v[0-9]\\.[0-9]{1,2}\\.[0-9]{1,2}$",
var.runs_on_version))
- error_message = "runs_on_version must be a RunsOn release tag such as
v3.5.1."
- }
-}
-
-variable "github_organization" {
- description = "GitHub organization the runners serve (case-sensitive)."
- type = string
- default = "apache"
-}
-
-variable "alert_email" {
- description = "Recipient of RunsOn cost reports and alerts; AWS sends an SNS
confirmation email on creation."
- type = string
- default = "[email protected]"
-}
-
-variable "enable_admin_routes" {
- description = <<-EOT
- Expose the /setup admin routes on the public entry point. Set to true only
while registering
- (or re-registering) the GitHub App, then back to false.
- EOT
- type = bool
- default = false
-}
-
-variable "vpc_cidr" {
- description = "CIDR of the VPC the stack creates; must not overlap existing
VPCs in the account (10.1.0.0/16 is taken)."
- type = string
- default = "10.2.0.0/16"
-}
diff --git a/terraform/runs-on/version.tf b/terraform/runs-on/version.tf
deleted file mode 100644
index f7cee6f..0000000
--- a/terraform/runs-on/version.tf
+++ /dev/null
@@ -1,27 +0,0 @@
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied. See the License for the
-# specific language governing permissions and limitations
-# under the License.
-
-terraform {
- required_providers {
- aws = {
- source = "hashicorp/aws"
- version = "~> 6.0"
- }
- }
- # S3-native state locking (use_lockfile) is GA from 1.11
- required_version = ">= 1.11.0"
-}
