Adding Bob back on cc in case he is not monitoring the dev list Andrew
On Fri, May 29, 2026 at 1:53 PM Oleks V. <[email protected]> wrote: > Thank you, Bob, for raising this concern regarding the consumption of ASF > shared resources. > > The Apache DataFusion team has already addressed several low-hanging > optimizations, and as a result, daily CI resource consumption has been > reduced by approximately 50%. 24Hr usage decreased from 113K minutes a week > ago to 55K minutes over the last 24 hours. > > These changes were introduced over the past few days, and the most recent > 5-day report is already approaching the threshold defined in the ASF policy: > https://infra.apache.org/github-actions-policy.html > > Specifically, the latest 5-day report shows 248K minutes compared to the > 216K threshold, with a clear downward trend continuing as the recent > optimizations fully propagate through the reporting window. > > In parallel, we are actively working on longer-term improvements that will > further reduce CI overhead while maintaining project quality and > contributor experience. > > Some of the options currently being evaluated include: > > * Offloading long-running CI pipelines to contributor forks, similar to > the approach used by Apache Spark: > https://github.com/apache/spark/tree/master/.github > This would shift execution of certain workflows from ASF-hosted > resources to contributor-owned resources. > * Refining test matrices to reduce redundant workflow combinations. > * Introducing a gated execution model where selected CI pipelines are > triggered manually instead of automatically for every commit. > > We would also appreciate any additional recommendations you may have based > on your experience managing CI resource utilization across ASF projects. > > > Given the significant reduction already achieved and the continuing > downward trend, would it be possible to extend the current deadline to > allow the team sufficient time to implement the longer-term optimizations > without negatively impacting project quality, contributor confidence and > user trust? > > We appreciate your support and look forward to your feedback. > > Oleks V > > On 2026/05/22 11:44:16 Robert Thomson wrote: > > Hello, Datafusion PMC. > > > > In 2024, the ASF introduced the policy for GitHub Actions usage > > across the foundation[1]. The ASF Github shared pool of > > Github-hosted runners has been at, or very close to the limit of > > 900 jobs most of the time in the past few weeks and this is the > > case again today. > > > > Your project has been identified as being among the top 5 consumers of > > build time over the past 7 days and we request that you bring your > > usage down by stream-lining long-running builds. Contact Infra for > > a consultation if you are unable to streamline your builds further. > > > > You can use the infra reporting tool[2] to monitor your GHA usage as you > > work on stream-lining, as well as locate any bottlenecks in the > workflows. > > > > Infra will allow you two weeks time (till the 8th of June, 2026) to > > progress this, but should you still be above the limits by then, > > without a viable path forward, we will be limiting your GHA usage. > > > > Kind regards, > > Bob Thomson, on behalf of ASF Infrastructure. > > > > > > [1] https://infra.apache.org/github-actions-policy.html > > [2] > > > https://infra-reports.apache.org/#ghactions&project=datafusion&hours=24&limit=15&group=name > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > >
