GitHub user alamb created a discussion: Request: reduce the amount of time used on Apache shared runners
TLDR: Could you please consider reducing the amount of github hosted runner time this project is consuming? 🙏 Hi, I am a member of the Apache DataFusion and Arrow PMCs. Our CI jobs that run on github hosted runners are taking hours to start due to other jobs running in the (shared) apache organization which is severely impacting our project's progress. While investigating what our jobs were waiting for, I was continually pointed at this repository. ## 117 **Runner-Days** of time in the last 24 hours, 20x more than next highest project I have a script that computes the total amount of github hosted runner time in hours spend for each project (attached below) The airflow project is currently using almost 20x more than the next highest project I can find, to the tune of almost 120 **days** of runner time in each 24 hour window. On average, at any time, Apache Airflow has 128 jobs running on the shared runners ```shell andrewlamb@Andrews-MacBook-Pro-3:~/Downloads$ python3 usage.py --time 24h apache/airflow 1535 runs of apache/airflow created since 2026-10-21T09:36:30Z (24.0 h window) jobs loaded: 1469 runs from cache, 66 from the API, 0 failed (cache: gh-runner-usage-cache) conclusion jobs share runner-min runner-days success 19020 58.7% 169845 117.9 skipped 10557 32.6% 0 0.0 cancelled 2417 7.5% 12016 8.3 failure 256 0.8% 2941 2.0 queued 70 0.2% 0 0.0 in_progress 69 0.2% 0 0.0 total 32389 184802 128.3 average concurrent jobs over the window: 128 ``` Here is the script (and the data to save you some GH API credits): [usage-airflow.zip](https://github.com/user-attachments/files/33813383/usage-airflow.zip) You can see more analysis / discussion on these issues - https://github.com/apache/datafusion/issues/25088 - ASF Slack Discussion: https://the-asf.slack.com/archives/CBX4TSBQ8/p1792520326623569 - https://issues.apache.org/jira/projects/INFRA/issues/INFRA-28560?filter=allopenissues I looked at this repo briefly to see if there was something obvious that was consuming all the time that maybe could reduce the time spent. I think it may just be that each commit runs significant amount of work on CI. I noticed that each commit in stacked PRs kicks off the entire CI suite, for example that I discussed with @ephraimbuddy in this PR) but I don't think this explains the entire usage - https://github.com/apache/airflow/pull/74793 cc @kaxil and @potiuk as I see your names on many approvals / PRs. Thank you for your consideration GitHub link: https://github.com/apache/airflow/discussions/75395 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
