gaogaotiantian opened a new pull request, #56798: URL: https://github.com/apache/spark/pull/56798
### What changes were proposed in this pull request? This backports the CI build-time optimization series from `branch-4.x`/`master` to `branch-4.1`. A `precompile` job builds Spark once and publishes the compile output as an artifact that the downstream matrix jobs consume (falling back to a local build if the precompile job is absent or fails), the per-job Coursier caches are unified under a single key, and the shared compile artifacts use zstd compression. Squashed backport of: - [SPARK-56768] Share SBT compile artifact across pyspark CI jobs - [SPARK-56831] Share SBT precompile artifact with sparkr CI job - [SPARK-56943] Share SBT precompile artifact with JVM build matrix - [SPARK-56964] Share Maven precompile artifact across maven_test matrix - [SPARK-57069] Share SBT precompile artifact with docker/k8s integration test CI jobs - [SPARK-57075] Share precompile Coursier cache with host-runner SBT jobs - [SPARK-57142] Share SBT precompile artifact with tpcds-1g CI job - [SPARK-57144] Unify Coursier cache to a single key across all jobs - [SPARK-56830] Share SBT compile artifact with python hosted runner CI jobs - [SPARK-57330] Switch shared CI compile artifacts to zstd compression Adaptations for `branch-4.1`: the Python toolchain stays on 3.11 with the branch's existing package pins, and the GitHub Actions are kept at the versions already pinned on `branch-4.1` (`actions/cache@v4`, `actions/cache/restore@v4`, `actions/checkout@v4`, `actions/setup-java@v4`, `actions/download-artifact@v4`, `actions/upload-artifact@v4`) rather than pulling in the unrelated action version bumps. As on `branch-4.x`, the `precompile` job is the sole Coursier cache writer and all consumer jobs are restore-only. ### Why are the changes needed? To cut redundant Scala/Maven compilation and Coursier cache duplication on `branch-4.1` CI, matching the optimization already present on the newer branches. ### Does this PR introduce _any_ user-facing change? No. CI-only. ### How was this patch tested? CI on this PR. The three workflow files validate with `python3 -c "import yaml; yaml.safe_load(...)"`. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (claude-opus-4-8) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
