Thanks Andrew, To give a bit more context, It now looks as follows (this is from a PR adds a new build to it, wasm32, taken from here <https://github.com/apache/arrow/actions/runs/408710849>):
[image: Screenshot 2020-12-08 at 18.54.36.png] The main consequences were: * build time halved in both cache hits and misses * reduced cache misses * code coverage is now part of the build (only ran in cron-job) * test of the c data interface against pyarrow is now part of the build (only ran in cron-job) * use only official rust docker images and stopped pulling and pushing custom images * easier to extend to variations of compiler flags, archs, compiler version, etc. The above was achieved through the following changes: * jobs that share artifacts share a common parent job (a common "need") and use github's cache action to share built artifacts * any independent work is assigned its own job, for increased parallelism (the reason for more jobs) * stopped using docker images to store intermediary state via github cache actions * stopped using archery and docker-compose to drive the execution. Instead, we run the official Rust docker images and use officially curated github actions to prepare the environment. The main build can be reproduced by a single command: docker run --rm -v $(pwd)/rust:/rust -it rust /bin/bash -c "cd /rust && cargo build" (this is notoriously slow on mac due to how docker volumes work). There are 3 items that can be further improved / optimized: * The integration test, that takes 20-30m, often re-builds entire artifacts even when no changes are made. I proposed something on a separate mailing thread to address this * Travis build and other github jobs are often triggered by rust-specific changes to `ci/` that do not affect them. * The lint step (rustfmt) currently runs together with all other implementations in a single job. It could be split off and run only to changes on rust. Best, Jorge On Tue, Dec 8, 2020 at 5:51 PM Andrew Lamb <[email protected]> wrote: > I just wanted to give everyone a heads up that we have merged an overhaul > of the Rust CI system. > > You can see more details and rationale on the PR: > https://github.com/apache/arrow/pull/8821 but I wanted to bring it to the > attention of a wider audience as the PR checks will look different for Rust > PRs. > > Thanks to @Jorge Cardoso Leitão <[email protected]> for driving > this work > > Andrew >
