(also adding dev@datafusion as they are affected as well) Hi Jarek,
For others who are looking for more context related to this email, there is an ASF slack thread[1] and issue[2] with more background and data. > And we are explaining that for weeks, yet, for some reason we get nudges from Andrew every day - even if he knows very well we are working on it. The reason I am nudging is that for the last several days, contributors in arrow-rs and datafusion are waiting 3-7 hours for CI jobs to start. I have heard frustration from many community members. This seems to have gotten much worse in the last week. While waiting for jobs to start, I started investigating what projects were using the runners that our jobs were waiting behind (the script I use is attached to issue [2]) which lead me to Airflow. I am sorry if it seems I am singling that project out but it seems to be using significantly more than the next largest project I can find. > (Airflow and Arrow) are two biggest users of the CI in ASF. Can you please share the data that you used to come to this conclusion? My numbers[3] suggest over the last 24 hours Airflow has used roughly 10x more shared runner time (106 runner-day vs 16 runner-days) than arrow/arrow-rs/arrow-go. > Andrew, are you actually working with other Arrow team members on applying those improvements I proposed (or other?). It seems a bit one sided when you have all proactiveness from our side, yet we do not hear what you are doing to help the ASF. I am not sure what you proposed as I don't use or follow Airflow; I can tell you that both arrow-rs and DataFusion communities have been busy trying to reduce our shared runner usage, I have listed a few examples below but you can find a bunch more if you search for recent merge PRs in those repositories Andrew [1]: https://the-asf.slack.com/archives/CBX4TSBQ8/p1792520326623569 [2]: https://github.com/apache/datafusion/issues/25088 [3]: https://github.com/apache/datafusion/issues/25088#issuecomment-6369916889 DataFusion: PRs to reduce shared runner load - https://github.com/apache/datafusion/pull/25591 (Oct 20) Runs the Rust workflow for fork PRs on the contributor's fork. - https://github.com/apache/datafusion/pull/26574 (Oct 21) Moves the circular dependency check onto the fork - https://github.com/apache/datafusion/pull/26571 (Oct 21) Moves about a dozen small jobs to ubuntu-slim. Arrow (Rust): PRs to reduce shared runner load - https://github.com/apache/arrow-rs/pull/11788: "Run fork PR test suites in contributor forks". - https://github.com/apache/arrow-rs/pull/11774: "ci: move several jobs to ubuntu-slim runner". - https://github.com/apache/arrow-rs/pull/11631: "Run Miri on a daily schedule, require it before releases". On Fri, Oct 23, 2026 at 12:27 PM Jarek Potiuk <[email protected]> wrote: > Hello here, > > I did not want to nudge anyone here, but since Andrew does it. Just > adding Andrew - who keeps on nudging Airflow in issues - we are working in > Airflow on implementing **way** more improvements than that - for several > weeks already - and we are close to conclusion. > And the only way to deal with queueing is to get big users (Airflow and > Arrow) are two biggest users of the CI in ASF. In Airflow we are working on > 80% cutting - via using self-hosted runners and via delegating CI to the > contributor's public runners. > > And we are explaining that for weeks, yet, for some reason we get nudges > from Andrew every day - even if he knows very well we are working on it. > > And - just to be sure - those 30% - 40% savings above are easily > achievable (way easier than the 80% of Airflow) - and it is also absolutely > necessary to remove the pressure from the queue., > > Andrew, > > Andrew, are you actually working with other Arrow team members on applying > those improvements I proposed (or other?). It seems a bit one sided when > you have all proactiveness from our side, yet we do not hear what you are > doing to help the ASF. > > I would really appreciate **some** answer here how are you working with > others in the Arrow to solve the issues. Are there any plans to improve > things ? > > And yes Andrew - I expect answer from you, > > J. > > On Thu, Oct 22, 2026 at 8:46 PM Benson Muite <[email protected]> > wrote: > >> Jarek Potiuk <[email protected]> writes: >> >> Hi, >> >> > Hi everyone, >> > >> > I hope you're all doing well! I've been looking at how Arrow uses >> > GitHub Actions. All ASF projects share the same pool of GitHub >> > runners, so every minute we save also shortens the queue for >> > everyone else. The good news: Arrow's CI is already in good shape >> > (path filters, concurrency cancelling, the "CI: Extra" labels, >> > RunsOn for CUDA). Even so, I found some easy wins, and I'd love to >> > hear what you think. >> > >> > I was a listener at yesterday's board meeting and i understand >> > that Arrow has some CI "difficulties" and is looking for ways to >> > deal with and JBO was generally offering his overlook of that >> > (hence the CC:) but then there was a lot of buzz about the >> > #project-workflows-optimisations channel of ASF - and I just added >> > new "workflow-optimization" skill in Magpie that is >> > producing very good advice, and you can use it to implement changes. >> > those optimizations quickly. >> > >> > Last week we used roughly 110-125k GitHub-hosted runner minutes. >> > Package Linux alone is about 40% of that. >> >> Fedora does offer CI builds for packaging: >> https://packit.dev/ >> >> Probably CentOS and other RPM builds could be moved to that. >> >
