potiuk opened a new pull request, #73: URL: https://github.com/apache/airflow-ci-infra/pull/73
The stack's Spot circuit breaker (RunsOn's default `2/15/30`) stops Spot launches for 30 minutes after 2 Spot interruptions within 15 minutes. With hundreds of concurrent runners, two interruptions in 15 minutes are routine, so the breaker stayed tripped and the stack ran mostly on on-demand instances. On 2026-10-26 it ran 920 on-demand vCPUs against 48 on Spot. The on-demand quota (2,000 vCPUs) reached about 93% around 17:08 UTC, and jobs queued for over an hour, for example `Min SQLAlchemy test` in apache/airflow run 40363018121. `30/15/10` keeps the breaker for a real Spot capacity crunch (30 interruptions in 15 minutes) and returns to Spot after 10 minutes. Interrupted jobs are re-run by RunsOn, and apache/airflow reports them on the PR (`spot-interruption-notice.yml`). Checks run locally: the prek hooks on the changed file. The plan of this PR shows the parameter change; it is applied from main after a required reviewer approves the `runs-on-prod` environment. --- ##### Was generative AI tooling used to co-author this PR? - [X] Yes — Claude Opus 5 Generated-by: Claude Opus 5 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
