This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch runs-on-v3.5.1-spot-capacity in repository https://gitbox.apache.org/repos/asf/airflow-ci-infra.git
commit d2fe138b224c53db449cacba4efb4cb68102d739 Author: Jarek Potiuk <[email protected]> AuthorDate: Tue Oct 27 21:39:22 2026 +0100 Upgrade RunsOn to v3.5.1 so a Spot capacity miss stops sending every job to on-demand Generated-by: Claude Opus 5 --- terraform/runs-on-access/main.tf | 2 +- terraform/runs-on/README.rst | 34 +++++++++++++++++++++++++++++++--- terraform/runs-on/runs-on.yml.example | 6 +++--- terraform/runs-on/variables.tf | 2 +- 4 files changed, 36 insertions(+), 8 deletions(-) diff --git a/terraform/runs-on-access/main.tf b/terraform/runs-on-access/main.tf index f1df0f0..771c71a 100644 --- a/terraform/runs-on-access/main.tf +++ b/terraform/runs-on-access/main.tf @@ -22,7 +22,7 @@ locals { # RunsOn releases the apply role may deploy; older published templates stay reachable otherwise. # Upgrading RunsOn means adding the new version here and applying this module first. - runs_on_versions = ["v3.3.2"] + runs_on_versions = ["v3.3.2", "v3.5.1"] # The repository is pinned by its immutable IDs (repository_owner_id and repository_id claims), # so a renamed or re-created repository cannot match. Fork pull requests get no OIDC token at all, diff --git a/terraform/runs-on/README.rst b/terraform/runs-on/README.rst index 1ff7026..e59290a 100644 --- a/terraform/runs-on/README.rst +++ b/terraform/runs-on/README.rst @@ -38,7 +38,7 @@ Files ``variables.tf`` All variables have defaults: ``stack_name`` (``runs-on``), ``github_organization`` (``apache``), ``alert_email`` (``[email protected]``), ``runs_on_version`` - (``v3.3.2``), ``enable_admin_routes`` (``false``) and ``vpc_cidr`` (``10.2.0.0/16``). The + (``v3.5.1``), ``enable_admin_routes`` (``false``) and ``vpc_cidr`` (``10.2.0.0/16``). The region (``us-east-2``) is not a variable: it is hard-coded in ``provider.tf``, the backend in ``state.tf`` and the commands below, so a move changes all of them. ``state.tf`` @@ -153,8 +153,11 @@ Rollback is the previous ``runs_on_version``. The apply role may only deploy the in ``runs_on_versions`` in `terraform/runs-on-access <../runs-on-access/README.rst>`_, so a ``runs_on_version`` bump first needs the new version added there and applied by an admin. For the same reason, rollback works only while the previous version is still listed: keep it during an -upgrade (for example ``["v3.3.2", "v3.4.0"]``) and drop it later. Template ``v3.3.2`` has -SHA-256 ``3017805d77c1d717fdd229a1e1242db12c985944699d3d8dca01f6f28b35c16d``. +upgrade (for example ``["v3.3.2", "v3.4.0"]``) and drop it later. Template SHA-256s: +``v3.3.2`` ``3017805d77c1d717fdd229a1e1242db12c985944699d3d8dca01f6f28b35c16d``, ``v3.5.1`` +``9acd6fd154e2bb921f95466c45380d2e3d7bd718553a12af4e5c76d15915bd57``. Both templates create the +same IAM roles, so the service role's guardrails in ``terraform/runs-on-access`` need no change +for this upgrade. Every plan shows ``"LicenseKey" = "****" -> "arn:...:parameter/runs-on/license-key"``: CloudFormation masks ``NoEcho`` parameters when Terraform reads the stack, and the AWS provider @@ -223,3 +226,28 @@ Things learned in the validation ``setup-*`` caches and requires ``runs-on/action`` before every ``upload-artifact``. - EC2 quotas in us-east-2 (``All Standard Spot Instance Requests`` 3000 vCPU, on-demand 2000) were enough for ~130 concurrent jobs. + +Spot capacity +------------- + +On 2026-10-25..27 the stack launched 4706 Spot and 4038 on-demand runners. The worker log +(``force_on_demand_reason`` on each "Using on-demand instances" line) gives the cause, and none +of it was the Spot quota (peak 2332 of 3000 vCPU); the on-demand quota filled up instead +(2004 of 2000 on 2026-10-26 16:00 UTC, 1653 ``VcpuLimitExceeded``): + +- ``spot_disabled`` (2641): the default ``SpotCircuitBreaker`` ``2/15/30`` kept Spot off; none + since the ``30/15/10`` apply on 2026-10-26 20:29 UTC. +- ``force_on_demand`` (678): x64 Spot capacity misses. Up to v3.4 one miss sends every job to + on-demand for five minutes; from v3.5.0 only the affected runners go. +- ``spot_recovery_rerun`` (719): RunsOn's rerun of an interrupted job, on-demand by design. + +Of 2156 Spot capacity errors in a day, 1959 were x64 (``m7a+m7i+m6a+m6i``) and 197 arm64. +More instance families give the Spot allocator more pools, so the x64 presets in +``runs-on.yml.example`` also allow ``m8a``, ``m8i``, ``m5a`` and ``m5`` (Spot ``2xlarge`` on +2026-10-27: 0.08-0.16 USD/hour against 0.12-0.15 for the existing families). Jobs that spell +the family out in their label (``apache/airflow`` workflows) need the same change there. + +The CloudFormation template places runners in two Availability Zones (us-east-2a and 2b, +fixed by the template). RunsOn's Spot spread across zones (``RUNS_ON_SPOT_FORCE_ZONE_SPREAD``, +v3.5.0 beta) and custom subnets exist only in its Terraform module, so a third zone means +moving off the CloudFormation stack. diff --git a/terraform/runs-on/runs-on.yml.example b/terraform/runs-on/runs-on.yml.example index 441061b..4f8b422 100644 --- a/terraform/runs-on/runs-on.yml.example +++ b/terraform/runs-on/runs-on.yml.example @@ -31,19 +31,19 @@ runners: airflow-amd-small: cpu: 4 ram: 16 - family: ["m7a", "m7i", "m6a", "m6i"] + family: ["m8a", "m8i", "m7a", "m7i", "m6a", "m6i", "m5a", "m5"] image: ubuntu22-full-x64 volume: 150gb airflow-amd-medium: cpu: 8 ram: 32 - family: ["m7a", "m7i", "m6a", "m6i"] + family: ["m8a", "m8i", "m7a", "m7i", "m6a", "m6i", "m5a", "m5"] image: ubuntu22-full-x64 volume: 150gb airflow-amd-xl: cpu: 32 ram: 128 - family: ["m7a", "m7i", "m6a", "m6i"] + family: ["m8a", "m8i", "m7a", "m7i", "m6a", "m6i", "m5a", "m5"] image: ubuntu22-full-x64 volume: 150gb airflow-arm-small: diff --git a/terraform/runs-on/variables.tf b/terraform/runs-on/variables.tf index 4f74e13..05d9fff 100644 --- a/terraform/runs-on/variables.tf +++ b/terraform/runs-on/variables.tf @@ -27,7 +27,7 @@ variable "stack_name" { variable "runs_on_version" { description = "RunsOn release to deploy; selects the version-pinned CloudFormation template." type = string - default = "v3.3.2" + default = "v3.5.1" } variable "github_organization" {
