Hi everyone, I hope you're all doing well! I've been looking at how Arrow uses GitHub Actions. All ASF projects share the same pool of GitHub runners, so every minute we save also shortens the queue for everyone else. The good news: Arrow's CI is already in good shape (path filters, concurrency cancelling, the "CI: Extra" labels, RunsOn for CUDA). Even so, I found some easy wins, and I'd love to hear what you think.
I was a listener at yesterday's board meeting and i understand that Arrow has some CI "difficulties" and is looking for ways to deal with and JBO was generally offering his overlook of that (hence the CC:) but then there was a lot of buzz about the #project-workflows-optimisations channel of ASF - and I just added new "workflow-optimization" skill in Magpie that is producing very good advice, and you can use it to implement changes. those optimizations quickly. Last week we used roughly 110-125k GitHub-hosted runner minutes. Package Linux alone is about 40% of that. Two things need attention soon: 1. The nightly failure reports have been silent since Oct 21. report_ci.yml installs "python-version: 3", which now gives Python 3.15. pygit2 has no wheel for 3.15 yet, so installing Archery fails and no email or Zulip report goes out. Pinning to 3.13 fixes it. 2. pull_request_target is being disabled for ASF repositories in November 2026. dev_pr.yml, pr_bot.yml and pr_limit.yml use it. PR Limit can simply become the .asf.yaml "creation_cap" setting, with the same limit of 3 open PRs per contributor and no workflow at all. Ideas to save minutes: - Stop running the Extra workflows (C++/R/CUDA Extra, Package Linux) on every merge to main, since they already run every night. This alone is about 25% of our weekly minutes, mostly Package Linux's 20 jobs of about 90 minutes each. - Once a PR has a "CI: Extra" label, every push reruns the full 20-job Package Linux matrix. Running a smaller subset on PRs (say amd64 only) and the full matrix nightly would save about another 10%. - Read the "CI: Extra" labels straight from the PR event instead of starting 2 small jobs per Extra workflow (about 870 jobs a week), and move the bot and labelling jobs to the new small "ubuntu-slim" runners. - Maybe: replace the full core run on each merge to main with a daily scheduled "canary" run (about 5%). - Housekeeping: max-parallel 15 on Package Linux and Python, timeouts on the bot jobs, cron schedules off minute 0. Altogether that's roughly 30%, or about 40% with the Package Linux PR subset. I could open PRs from my Magpie skills, but I do not know Arrow well enough to know which of those make sense, so you might want to implement those changes yourself. How I did it: I ran the new "workflow-optimization" skill from Apache Magpie (https://magpie.apache.org/), in the repo-health family of skills. It works with the AI agent you already use. It reads a repository's workflows, .asf.yaml, Dependabot config and a week of Actions run history, and checks them against Infra's upcoming "Optimizing GitHub Actions workflows" guide (in review at https://github.com/apache/infrastructure-website/pull/320). The result is a prioritised plan where every finding points to a file and line and the numbers behind it. The skill only reads; it changes nothing in the repository. You can try it yourself: 1. Follow the install guide: https://github.com/apache/magpie/blob/main/docs/setup/marketplace-install.md 2. In Claude Code, install the skills: /plugin install magpie-repo-health@apache-magpie 3. Run it in your Arrow checkout: /magpie-repo-health:workflow-optimization (or just ask your agent to "optimize our CI workflows") When you run it with your agent, it ends by suggesting how to split the findings into separate PRs, and offers to draft each one with you, one at a time. You review everything before anything is pushed. Jean-Baptiste, I've CC'd you since you're on both the Arrow and Magpie sides. Feedback on the skill is very welcome! Thanks, and have a great day! Jarek -- Analysis and draft prepared with help from Claude Code using the Apache Magpie workflow-optimization skill; reviewed by me.
