Thanx Robert, Yufei and JB for looking into it. As suggested, I tried to look into which Jobs can be migrated to ubuntu-slim
it is a small group of short checks * Can be moved with no change - required-checks (ci.yml) — a single bash conditional over needs.*.result, no checkout. It exists only to give branch protection one status to gate on. https://github.com/ayushtkn/polaris/actions/runs/34565642910/job/103160643626 - stale (stale.yml) — one pinned JS action (actions/stale) for issue/PR housekeeping, on a daily schedule. https://github.com/ayushtkn/polaris/actions/runs/34574237202/job/103182977196 - codeql (ci-security.yml) — runs with languages: actions, so it extracts workflow YAML only; no compiler, no build, no container. https://github.com/ayushtkn/polaris/actions/runs/34576020059/job/103188534433 The ref commit: https://github.com/ayushtkn/polaris/commit/849c82767e56b46b35b2f2d815d9cfe21ee7d9c5 * Can be moved with some changes - zizmor (ci-security.yml) — zizmorcore/zizmor-action runs the tool through docker pull + docker run and aborts with "Cannot run this action without Docker"; a slim job is itself an unprivileged container with no daemon. Replaced with the CLI via uv, same arguments. Both branches validated: annotations (pull_request): https://github.com/ayushtkn/polaris/actions/runs/34576020059/job/103188534246 SARIF + GHAS upload (push): https://github.com/ayushtkn/polaris/actions/runs/34581371648/job/103205453383 The ref commit: https://github.com/ayushtkn/polaris/commit/d4704989ae2e257f88d8fa172fdfd9a5cb1e7910 Maybe I can improve more here... - prerequisite-checks, generate-release-email (release-3), create-release-branch (release-1) — git and bash only. Each also drops setup-test-env, which writes a testcontainers config and a docker.io mirror and then restarts the Docker daemon; none of the three runs a container, and the restart cannot work unprivileged. https://github.com/ayushtkn/polaris/actions/runs/34571681760/job/103174954957 https://github.com/ayushtkn/polaris/actions/runs/34571681760/job/103175083529 https://github.com/ayushtkn/polaris/actions/runs/34572044132/job/103176054604 The ref commit: https://github.com/ayushtkn/polaris/commit/5353bc06ba87c49cb5177f50101ff70be0d41d3b * Can not be moved or which have challenges The nine Gradle/Java jobs stay: ubuntu-latest preinstalls JDK 21, so setup-java is a 0.0s toolcache hit today, and on slim it would download a JDK every run. docker-image-build, docker-image-scan, regtest, site, python-client and helm-tests need a Docker daemon or minikube. markdown-link-check stays because tcort/github-action-markdown-link-check is itself a container action (runs: using: docker). cancel-release-candidate stays because it does sudo apt-get install subversion. The change I observed: codeql goes 39s → 93s (ubuntu-24.04 preinstalls a CodeQL bundle, slim does not, and analysis runs on one core), while zizmor is unchanged at 19s → 20s Concretely, using last week's run counts (107 ci-pr, 30 ci-main, 138 ci-security) and measured job durations: Polaris consumes ~18,100 standard-runner minutes a week (~302 h). This moves ~147 of them (~2.5 h, 0.8%), but takes ~413 job starts a week out of the shared queue — notably the entire Security workflow, which runs 138 times a week and stops contending altogether. The five longest job types are 67% of the total on their own and none of them can move, so this is a queue-pressure change rather than a capacity one.. I know the savings for us it isn't that impressive and won't solve the problem altogether what Yufei flagged, but I thought to just do a quick try, so that we have a reference point atleast. PS. The Github Action runs are on my fork on top of a temp commit to make actions run on the fork. -Ayush On 2026/09/10 09:10:42 Robert Stupp wrote: > Hi Yufei, > > Thanks for raising this. > > Thanks, Ayush, for digging into this and sharing the shared-queue > explanation and the relevant reporting/tooling links. That gives us a > useful factual basis for deciding what Polaris can improve. > > The queue is shared across ASF, so staying below an individual project's > policy ceiling is necessary but should not be the only goal. It does not by > itself remove pressure from a saturated pool. > > I do not expect ubuntu-slim to materially help Polaris. Its 15-minute limit > and restricted environment mean it is likely suitable only for a small > group of short checks, not the longer test jobs. > > Could you identify the specific Polaris workflows or jobs that you think > are suitable, and either open a focused PR or summarize the expected > runner-minute saving? That would let us evaluate a concrete improvement > against test coverage and reliability. > > Regards, > Robert > > On Thu, Sep 10, 2026 at 7:53 AM Yufei Gu <[email protected]> wrote: > > > Hi Ayush, thanks for looking into this. We could also move some of our CI > > workflows to ubuntu-slim. It has following limits per PR > > https://github.com/apache/airflow/pull/72852: > > > > ubuntu-slim's constraints, verified against the runner image rather than > > > assumed: > > > > > > 1. No Docker daemon — client only, unprivileged, no DinD. Everything > > > touching Breeze, image builds or Docker-container actions stays put. > > > 2. python3 is Ubuntu's system interpreter — PEP 668 externally > > > managed, so a bare pip install fails. > > > 3. Minimal toolset — git, gh, jq, node, curl present; no Java, no Go. > > > actions/setup-* still works (tool-cache downloads). > > > 4. One core — jobs doing real work need timeout headroom. > > > > > > > > Yufei > > > > > > On Wed, Sep 9, 2026 at 5:57 PM Ayush Saxena <[email protected]> > > wrote: > > > > > Hi Yufei, > > > I tried to grab some information around it, the best we have around per > > > project is this report [1] from Apace Infra, It shows the usage to > > people, > > > if they are part of that project. > > > > > > Regarding the queue, it is not per project it is at ASF level, the > > response > > > ``` > > > there is no per-project budgeting. One giant pool, and everybody dirties > > > it. the infra-reports is the best available, given limited introspection > > > available from GitHub.com > > > the short answer is that all the runner queues are full, despite our > > large > > > queue slot allocation > > > ``` > > > > > > I even while lurking around found some projects are trying to move to > > > ubuntu-slim for jobs which are small and takes less time <15mins to avoid > > > queuing ubuntu-latest like here: > > > https://github.com/apache/magpie/pull/1190 > > > https://github.com/apache/airflow/pull/72852 > > > > > > So, maybe we can also move a few of our jobs to ubuntu-slim, the > > reasoning > > > mentioned to move to ubuntu-slim was that no job can run on it more than > > 15 > > > mins and chances of runners blocked by some huge job isn't possible > > because > > > it hard aborts at 15 minutes. > > > > > > There is an Infra script added here [2] for watching out who is using how > > > much in case of such outages, but nothing around to solve it yet. > > > > > > That is the best I could find :-) > > > > > > -Ayush > > > > > > [1] https://infra-reports.apache.org/#ghactions > > > [2] https://github.com/apache/infrastructure-actions/pull/1259 > > > > > > On 2026/09/09 21:11:00 Yufei Gu wrote: > > > > Hi all, > > > > > > > > Polaris GitHub Actions workflows are experiencing long queue times, > > > > affecting multiple PRs and main-branch CI today (Sept. 9). Eight PR CI > > > runs > > > > were queued and two were still in progress. > > > > > > > > One main-branch CI run waited roughly four hours before its initial > > jobs > > > > started. It now has 10 successful jobs, but another 10 have been > > waiting > > > > for runners for about 27 minutes: > > > > https://github.com/apache/polaris/actions/runs/34377371899 > > > > > > > > Similar queueing was also visible in other Apache projects earlier > > today. > > > > ASF shared runner capacity may be a factor, but I haven’t confirmed > > > whether > > > > the organization’s concurrency limit is currently reached. > > > > > > > > Is anyone aware of an ongoing Infra issue, or able to help confirm the > > > > current shared runner capacity? > > > > > > > > Thanks, > > > > Yufei > > > > > > > > > >
