Fewer parallel runs is essential yes - we are at 900/900 GitHub hosted runner jobs/slots just now and looking at Superset Actions we can see nearly 500 completed Supeset repo action runs in the last hour, some of those are up to 25 minutes in execution time, so anything that can be done to reduce the share of runner jobs used by Superset is an urgent issue when we are at max jobs on runners on a daily basis now.
Thanks. Kind regards, -Bob Thomson, ASF Infrastructure On 2026/05/22 19:54:16 Evan Rusackas wrote: > Hi Bob (and everyone here), > > Thanks for the alert. The unfortunate thing is that this will only get worse > as we create/fix more things (security, dependabot, etc). Things only seem to > be ramping up. > > So, agreed, we must whittle. Cypress is the obvious killer (about half the > consumption). We’ll try to find ways to whittle away at this (we’re migrating > to Playwright, but it takes time). We might also be able to spend less > compute and more time by optimizing (or removing) some parallelization here. > > We’re also looking at moving from dependabot for all dependency bumps (a LOT > of PRs) to `renovate` - which might optimize things a bit (bumping > dependencies in groups) but we will need to also leave dependabot in place > for security-driven fixes as well. > > As for Cypress tests, we have some “martixification” happening, that I think > we can optimize. For the Superset folks reading this, I think we can split > out the “app_root” tests to JUST run on merges to `master` rather than every > PR. That’ll save ~50% right there, we just have to keep a better eye on CI on > `master` (which we haven’t been great about historically, but we’re getting > better). > > Here’s the app_root PR https://github.com/apache/superset/pull/40385 > > We can also reduce the E2E parallelization shards from 6 to… I dunno… 3 or 4. > That’ll save a fair bit of setup time spinning up Superset instances. Tests > will run a bit longer, but consume less overall. Seems like a fair tradeoff. > > Open to other ideas… maybe running fewer GHA workflows in parallel, and > having things more sequentially to fail faster (like nothing runs until > pre-commit passes, for example). > > Also, least importantly, we don’t have the access to see how we stack up > against other projects, but I sure am curious. > > Anyone's thoughts/PRs welcomed. > > Evan Rusackas > Preset | preset.io > On May 22, 2026 at 4:46 AM -0700, Robert Thomson <[email protected]>, wrote: > > Hello, Superset PMC. > > > > In 2024, the ASF introduced the policy for GitHub Actions usage > > across the foundation[1]. The ASF Github shared pool of > > Github-hosted runners has been at, or very close to the limit of > > 900 jobs most of the time in the past few weeks and this is the > > case again today. > > > > Your project has been identified as being among the top 5 consumers of > > build time over the past 7 days and we request that you bring your > > usage down by stream-lining long-running builds. Contact Infra for > > a consultation if you are unable to streamline your builds further. > > > > You can use the infra reporting tool[2] to monitor your GHA usage as you > > work on stream-lining, as well as locate any bottlenecks in the workflows. > > > > Infra will allow you two weeks time (till the 8th of June, 2026) to > > progress this, but should you still be above the limits by then, > > without a viable path forward, we will be limiting your GHA usage. > > > > Kind regards, > > Bob Thomson, on behalf of ASF Infrastructure. > > > > > > [1] https://infra.apache.org/github-actions-policy.html > > [2] > > https://infra-reports.apache.org/#ghactions&project=superset&hours=24&limit=15&group=name >
