GitHub user abnobdoss created a discussion: Streamlining CI: reducing job count and GHA cache pressure
I'd like to propose a couple of ways we could improve and streamline CI, though I completely understand if there isn't the appetite for structural changes right now. If there's interest, I'd bring them in as small, independently reviewable PRs. ## The problem `ci.yml` runs about 12 jobs on every PR, six of them some flavor of "build", and a lot of them recompile the same dependency graph independently. It works, but it's slower than it needs to be, hard to read at a glance, and it's pushed us past GitHub's 10 GB per-repo cache cap (we're at ~10.7 GB and already evicting). That ceiling only gets tighter as things like the integration tests grow toward what the other Iceberg subprojects run. Both come back to the same dependencies being compiled several times over. The two proposed changes below each take a piece of that: one on the build jobs, one on the check/lint steps. ## Consolidating the builds For the builds, I'd like to propose folding each OS's separate jobs into one, so the dependencies compile once per OS. Today each OS runs `build` and `build_with_no_default_features`, and Ubuntu also runs `check_standalone` and a separate `tests` job. That would look like: | New job | What it runs | Replaces today | |---|---|---| | `build-and-test` (Ubuntu) | build, `--no-default-features`, per-crate standalone check, nextest, doc tests | `build`, `build_with_no_default_features`, `check_standalone`, `tests` on Ubuntu | | `build` (macOS, Windows) | build, `--no-default-features` | `build`, `build_with_no_default_features` on macOS/Windows | | `msrv` (Ubuntu) | MSRV check | unchanged | Everything that runs today would still run; the `--no-default-features` build would become a step in the build job, so that coverage is kept. It would also be the main cache win. The Ubuntu `build`, `tests`, and `check_standalone` caches are each around 1.1 GB and are mostly the same dependency closure under different names. Folding them into one entry per OS would get a good chunk of that 10 GB back. ## Splitting up the check job For the check job, I'd like to propose splitting its two kinds of work. It currently bundles quick checks that don't compile anything (formatting, license header, machete, and so on) with clippy, which compiles the whole dependency graph. Pulling the non-compiling checks into their own `lint` job: | New job | What it runs | Replaces today | |---|---|---| | `lint` (Ubuntu) | license header, taplo fmt, cargo fmt, `Cargo.lock` check, cargo-machete, typos | `check`'s non-clippy steps, plus the separate Typos workflow | With nothing to compile, `lint` would fail a formatting nit or typo in seconds instead of behind a full build. We could also install taplo and cargo-machete as prebuilt binaries via `taiki-e/install-action` (already used for cargo-nextest) instead of building them from source each run, saving ~2.5 minutes, and pull in the typos check that currently sits in its own workflow. In this proposal, clippy would stay its own job, but run on Ubuntu only. Today it runs on both, and the macOS run is the single longest job on a green PR at around 10-12 minutes, versus ~6-7 on Ubuntu, almost all of it clippy. No code in the repo is gated on `target_os`/`unix`/`windows`, so clippy's output is identical on both and the macOS run is just redundant lint work (the compile coverage there stays via `build`). Dropping it takes that 10-12 minute job off the critical path, and since it's the one change that removes a per-platform run, it's the piece I'd most want input on. ## Feedback welcome Mainly looking for thoughts on: - Does the build consolidation seem reasonable? - Same for splitting the lint steps out from clippy. - On clippy: is running it on Ubuntu only acceptable, given there's no platform-specific code to lint? Happy to adjust any of this. GitHub link: https://github.com/apache/iceberg-rust/discussions/2753 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
