Thanks, Ayush, for looking into this and testing the changes. Given the limited impact, I think we can hold off for now. Having different runner environments and extra setup to maintain adds some complexity, and I’m not sure the benefit is enough to justify it yet. If the queue delays come back, we can revisit this. Your findings will be helpful then.
Yufei On Fri, Sep 11, 2026 at 3:02 AM Ayush Saxena <[email protected]> wrote: > Thanx Robert, Yufei and JB for looking into it. As suggested, I tried to > look into which Jobs can be migrated to ubuntu-slim > > it is a small group of short checks > > * Can be moved with no change > > - required-checks (ci.yml) — a single bash conditional over > needs.*.result, no checkout. It exists only to give branch protection one > status to gate on. > > https://github.com/ayushtkn/polaris/actions/runs/34565642910/job/103160643626 > - stale (stale.yml) — one pinned JS action (actions/stale) for issue/PR > housekeeping, on a daily schedule. > > https://github.com/ayushtkn/polaris/actions/runs/34574237202/job/103182977196 > - codeql (ci-security.yml) — runs with languages: actions, so it > extracts workflow YAML only; no compiler, no build, no container. > > https://github.com/ayushtkn/polaris/actions/runs/34576020059/job/103188534433 > > The ref commit: > https://github.com/ayushtkn/polaris/commit/849c82767e56b46b35b2f2d815d9cfe21ee7d9c5 > > * Can be moved with some changes > - zizmor (ci-security.yml) — zizmorcore/zizmor-action runs the tool > through docker pull + docker run and aborts with "Cannot run this action > without Docker"; a slim job is itself an unprivileged container with no > daemon. Replaced with the CLI via uv, same arguments. Both branches > validated: > annotations (pull_request): > https://github.com/ayushtkn/polaris/actions/runs/34576020059/job/103188534246 > SARIF + GHAS upload (push): > https://github.com/ayushtkn/polaris/actions/runs/34581371648/job/103205453383 > > The ref commit: > https://github.com/ayushtkn/polaris/commit/d4704989ae2e257f88d8fa172fdfd9a5cb1e7910 > Maybe I can improve more here... > > - prerequisite-checks, generate-release-email (release-3), > create-release-branch (release-1) — git and bash only. Each also drops > setup-test-env, which writes a testcontainers config and a docker.io > mirror and then restarts the Docker daemon; none of the three runs a > container, and the restart cannot work unprivileged. > > https://github.com/ayushtkn/polaris/actions/runs/34571681760/job/103174954957 > > https://github.com/ayushtkn/polaris/actions/runs/34571681760/job/103175083529 > > https://github.com/ayushtkn/polaris/actions/runs/34572044132/job/103176054604 > > The ref commit: > https://github.com/ayushtkn/polaris/commit/5353bc06ba87c49cb5177f50101ff70be0d41d3b > > * Can not be moved or which have challenges > > The nine Gradle/Java jobs stay: ubuntu-latest preinstalls JDK 21, so > setup-java is a 0.0s toolcache hit today, and on slim it would download a > JDK every run. docker-image-build, docker-image-scan, regtest, site, > python-client and helm-tests need a Docker daemon or minikube. > markdown-link-check stays because tcort/github-action-markdown-link-check > is itself a container action (runs: using: docker). > cancel-release-candidate stays because it does sudo apt-get install > subversion. > > The change I observed: codeql goes 39s → 93s (ubuntu-24.04 preinstalls a > CodeQL bundle, slim does not, and analysis runs on one core), while zizmor > is unchanged at 19s → 20s > > Concretely, using last week's run counts (107 ci-pr, 30 ci-main, 138 > ci-security) and measured job durations: Polaris consumes ~18,100 > standard-runner minutes a week (~302 h). This moves ~147 of them (~2.5 h, > 0.8%), but takes ~413 job starts a week out of the shared queue — notably > the entire Security workflow, which runs 138 times a week and stops > contending altogether. The five longest job types are 67% of the total on > their own and none of them can move, so this is a queue-pressure change > rather than a capacity one.. I know the savings for us it isn't that > impressive and won't solve the problem altogether what Yufei flagged, but I > thought to just do a quick try, so that we have a reference point atleast. > > PS. The Github Action runs are on my fork on top of a temp commit to make > actions run on the fork. > > -Ayush > > On 2026/09/10 09:10:42 Robert Stupp wrote: > > Hi Yufei, > > > > Thanks for raising this. > > > > Thanks, Ayush, for digging into this and sharing the shared-queue > > explanation and the relevant reporting/tooling links. That gives us a > > useful factual basis for deciding what Polaris can improve. > > > > The queue is shared across ASF, so staying below an individual project's > > policy ceiling is necessary but should not be the only goal. It does not > by > > itself remove pressure from a saturated pool. > > > > I do not expect ubuntu-slim to materially help Polaris. Its 15-minute > limit > > and restricted environment mean it is likely suitable only for a small > > group of short checks, not the longer test jobs. > > > > Could you identify the specific Polaris workflows or jobs that you think > > are suitable, and either open a focused PR or summarize the expected > > runner-minute saving? That would let us evaluate a concrete improvement > > against test coverage and reliability. > > > > Regards, > > Robert > > > > On Thu, Sep 10, 2026 at 7:53 AM Yufei Gu <[email protected]> wrote: > > > > > Hi Ayush, thanks for looking into this. We could also move some of our > CI > > > workflows to ubuntu-slim. It has following limits per PR > > > https://github.com/apache/airflow/pull/72852: > > > > > > ubuntu-slim's constraints, verified against the runner image rather > than > > > > assumed: > > > > > > > > 1. No Docker daemon — client only, unprivileged, no DinD. > Everything > > > > touching Breeze, image builds or Docker-container actions stays > put. > > > > 2. python3 is Ubuntu's system interpreter — PEP 668 externally > > > > managed, so a bare pip install fails. > > > > 3. Minimal toolset — git, gh, jq, node, curl present; no Java, no > Go. > > > > actions/setup-* still works (tool-cache downloads). > > > > 4. One core — jobs doing real work need timeout headroom. > > > > > > > > > > > Yufei > > > > > > > > > On Wed, Sep 9, 2026 at 5:57 PM Ayush Saxena <[email protected]> > > > wrote: > > > > > > > Hi Yufei, > > > > I tried to grab some information around it, the best we have around > per > > > > project is this report [1] from Apace Infra, It shows the usage to > > > people, > > > > if they are part of that project. > > > > > > > > Regarding the queue, it is not per project it is at ASF level, the > > > response > > > > ``` > > > > there is no per-project budgeting. One giant pool, and everybody > dirties > > > > it. the infra-reports is the best available, given limited > introspection > > > > available from GitHub.com > > > > the short answer is that all the runner queues are full, despite our > > > large > > > > queue slot allocation > > > > ``` > > > > > > > > I even while lurking around found some projects are trying to move to > > > > ubuntu-slim for jobs which are small and takes less time <15mins to > avoid > > > > queuing ubuntu-latest like here: > > > > https://github.com/apache/magpie/pull/1190 > > > > https://github.com/apache/airflow/pull/72852 > > > > > > > > So, maybe we can also move a few of our jobs to ubuntu-slim, the > > > reasoning > > > > mentioned to move to ubuntu-slim was that no job can run on it more > than > > > 15 > > > > mins and chances of runners blocked by some huge job isn't possible > > > because > > > > it hard aborts at 15 minutes. > > > > > > > > There is an Infra script added here [2] for watching out who is > using how > > > > much in case of such outages, but nothing around to solve it yet. > > > > > > > > That is the best I could find :-) > > > > > > > > -Ayush > > > > > > > > [1] https://infra-reports.apache.org/#ghactions > > > > [2] https://github.com/apache/infrastructure-actions/pull/1259 > > > > > > > > On 2026/09/09 21:11:00 Yufei Gu wrote: > > > > > Hi all, > > > > > > > > > > Polaris GitHub Actions workflows are experiencing long queue times, > > > > > affecting multiple PRs and main-branch CI today (Sept. 9). Eight > PR CI > > > > runs > > > > > were queued and two were still in progress. > > > > > > > > > > One main-branch CI run waited roughly four hours before its initial > > > jobs > > > > > started. It now has 10 successful jobs, but another 10 have been > > > waiting > > > > > for runners for about 27 minutes: > > > > > https://github.com/apache/polaris/actions/runs/34377371899 > > > > > > > > > > Similar queueing was also visible in other Apache projects earlier > > > today. > > > > > ASF shared runner capacity may be a factor, but I haven’t confirmed > > > > whether > > > > > the organization’s concurrency limit is currently reached. > > > > > > > > > > Is anyone aware of an ongoing Infra issue, or able to help confirm > the > > > > > current shared runner capacity? > > > > > > > > > > Thanks, > > > > > Yufei > > > > > > > > > > > > > > >
