namanjain24-sudo commented on issue #25148: URL: https://github.com/apache/datafusion/issues/25148#issuecomment-5641221263
@adriangb you are right about parallel steps and my "folding them into one job serialises their check steps" was out of date — `parallel`, `background`, `wait`/`wait-all` and `cancel` shipped on 2026-06-25, so one job can absolutely run the seven concurrently. That part of my answer was wrong. It still does not buy what the job split buys, for a reason that is not about scheduling: every step of a job runs on that job's single runner, so seven concurrent `cargo check` steps would share one machine's cores rather than getting seven machines. These steps do use the cores they are given. Today's `rust.yml` runs (18 `pull_request`, 7 `merge_group`) happen to contain a natural experiment, because four of the seven move to a 16-vCPU `runs-on` runner outside `pull_request` and three are pinned to `ubuntu-latest` everywhere. Median minutes: | job | PR (`ubuntu-latest`) | merge queue | runner outside PR | | --- | ---: | ---: | --- | | `cargo check datafusion features` | 7.07 | 5.33 | `runs-on cpu=16` | | `cargo check datafusion-substrait features` | 8.65 | 3.12 | `runs-on cpu=16` | | `cargo check datafusion-proto features` | 3.77 | 2.63 | `runs-on cpu=16` | | `cargo check datafusion-ffi features` | 2.36 | 1.12 | `runs-on cpu=16` | | `cargo check datafusion-common features` | 2.04 | 2.02 | `ubuntu-latest` | | `cargo check datafusion-functions features` | 2.71 | 2.70 | `ubuntu-latest` | | `cargo check datafusion-spark features` | 3.06 | 3.22 | `ubuntu-latest` | The bottom three are the control: they stay on `ubuntu-latest` in both events and their timings do not move between events at all, so the top four are not getting faster because of anything about the merge queue. They get faster when the machine changes. Reading more than that out of it would be overclaiming, since runner class, image and cache all change together — only `datafusion` and `datafusion-substrait` declare a `Rust Dependency Cache` step at all. But it is enough to say these are compile steps competing for CPU, not steps idling on each other, which is the case where in-job parallelism pays. The "build the dependency graph once" half of your question is the part that is genuinely unclaimed today, and it needs no restructuring: five of the seven declare no cache step whatsoever, so each builds the overlapping graph cold. Giving them the `shared-key: amd-ci` the other two already use is a much smaller change than merging the jobs, and it is independent of where they run. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
