jorgecarleitao opened a new pull request #8821: URL: https://github.com/apache/arrow/pull/8821
This PR refactors the CI with the purpose of making it faster, easier to extend, and more robust. The main consequences of this PR are: 1. build time of non-integration jobs reduced by half (11m vs 20m) 2. coverage is now part of all builds 3. significantly reduced the risk of cache misses, and the size of the caches 4. macos now have caching in place, like other builds 5. significantly easier to extend to other architectures, os, etc. The overall design of this PR is that most steps of the CI process should be independent of all others as much as possible, so that * they can run in parallel * it is clearer where the failure is * it is easier to extend the build to other variations (arch, os, flags, etc.) Intrinsically, the CI pipeline is just a DAG where certain jobs benefit from shared compute and thus depend on common nodes, while others run independently. This PR re-writes most of our CI as simple github jobs that share caches whenever makes sense, thereby describing the aforementioned DAG. Consequently, many of the sequential steps are now executed in parallel, causing the items describe above. There are two tasks depending on rust that are not in the `rust.yaml`: * lint, that happens via archery * integration tests, that happens via archery The bottleneck of a green pipeline are the integration tests, that currently re-compile a significant number of components (as I already flagged on the mailing list), but at least we get faster failures for compilation (2.5m) and clippy (4.5m). ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
