This is an automated email from the ASF dual-hosted git repository.
HyukjinKwon pushed a commit to branch branch-4.2
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.2 by this push:
new 496b82bf9bd3 [SPARK-56768][PYTHON][INFRA] Share SBT compile artifact
across pyspark CI jobs
496b82bf9bd3 is described below
commit 496b82bf9bd3a2101886bc764bc941e20a06d18a
Author: Ruifeng Zheng <[email protected]>
AuthorDate: Fri May 8 18:54:02 2026 +0900
[SPARK-56768][PYTHON][INFRA] Share SBT compile artifact across pyspark CI
jobs
### What changes were proposed in this pull request?
This PR adds a single shared `precompile` CI job that runs Spark's SBT
build once and uploads the resulting `target/` trees as a GitHub Actions
artifact. The 8 pyspark matrix entries plus the optional `pyspark-install`
entry now consume that artifact instead of re-running the same SBT build
themselves. The job is named generically because the same artifact can be
reused by sparkr, R, or documentation jobs in follow-ups.
Concretely:
- New `precompile` job in `.github/workflows/build_and_test.yml` runs the
SBT build:
```
./build/sbt -Phadoop-3 -Pyarn -Pspark-ganglia-lgpl -Phadoop-cloud -Phive \
-Pkubernetes -Pjvm-profiler -Pkinesis-asl -Phive-thriftserver \
-Pdocker-integration-tests -Pvolcano \
Test/package streaming-kinesis-asl-assembly/assembly connect/assembly
assembly/package
```
It tars every `target/` directory (excluding `./build/` and `./.git/`)
with `tar -czf` (gzip), uploads as `spark-compile-<branch>-<run_id>` with
`retention-days: 1` so storage is reclaimed within 24h.
The job's `if:` gate fires when any of `pyspark`, `pyspark-pandas`, or
`pyspark-install` is true in the precondition output, so the artifact is always
available for any matrix entry that needs it (including via `inputs.jobs`
overrides used by scheduled / dispatched workflows).
- The `pyspark` matrix job adds `precompile` to `needs:`, downloads and
extracts the artifact (with graceful fallback - see below), and exports
`SKIP_SCALA_BUILD=true` for `dev/run-tests.py` only when the artifact was
successfully extracted.
- `dev/run-tests.py` reads `SKIP_SCALA_BUILD` and skips
`build_apache_spark` and `build_spark_assembly_sbt` when it equals `"true"`,
matching the existing `SKIP_PACKAGING` idiom in the same file.
### Optional: graceful fallback if precompile fails
The precompile job is treated as an optimization, not a hard prerequisite:
- `precompile` has `continue-on-error: true` - a failed or cancelled
precompile does not fail the workflow run.
- The pyspark matrix's "Download precompiled artifact" step is gated on
`needs.precompile.result == 'success'` and itself has `continue-on-error: true`.
- The "Extract precompiled artifact" step is gated on the download
succeeding, and also has `continue-on-error: true`.
- Inside the "Run tests" bash block, `SKIP_SCALA_BUILD=true` is exported
only when `steps.extract-precompiled.outcome == 'success'`. Otherwise it stays
unset and `dev/run-tests.py` falls back to the original local SBT build.
So the worst case for a precompile failure is degraded to the pre-PR
behavior, not a workflow failure.
### SBT invocations: before vs. after
Every pyspark matrix entry today drives `dev/run-tests.py`, which makes two
SBT calls back-to-back (`build_spark_sbt` at `dev/run-tests.py:647` then
`build_spark_assembly_sbt` at `dev/run-tests.py:656`):
```
# build_spark_sbt
./build/sbt <11 profiles> Test/package
streaming-kinesis-asl-assembly/assembly connect/assembly
# build_spark_assembly_sbt
./build/sbt <11 profiles> assembly/package
```
The 11 profiles, identical across all 8 entries: `-Phadoop-3 -Pyarn -Phive
-Phive-thriftserver -Pkubernetes -Phadoop-cloud -Pjvm-profiler
-Pspark-ganglia-lgpl -Pkinesis-asl -Pdocker-integration-tests -Pvolcano`.
After this PR, when the artifact is usable, both calls are gated off via
`SKIP_SCALA_BUILD=true`; no SBT compile runs in the matrix entry. The new
`precompile` job runs one SBT invocation that combines all four goals (safe
because `SKIP_MIMA=true` in the pyspark job, so the original split for
`dev/mima` is moot here):
| | SBT compile invocations per pyspark matrix entry | Total SBT compile
invocations across the matrix |
|---|--:|--:|
| Before | 2 | 16 |
| After | 0 (artifact reused) or 2 (fallback) | 1 (in `precompile`, all 4
goals combined), plus 0 or 16 (fallback) |
The produced `target/` is byte-equivalent - same goals, same profiles, same
Scala/Java/Hadoop versions.
### Measured savings
Comparing two real CI runs of the same workflow on the same fork:
- **BEFORE** ([run
25432660319](https://github.com/zhengruifeng/spark/actions/runs/25432660319),
without this PR)
- **AFTER** ([run
25532354044](https://github.com/zhengruifeng/spark/actions/runs/25532354044),
with this PR active; pyspark-only iteration)
Per-matrix-entry wall time:
| Matrix entry | Before | After | Saved |
|---|--:|--:|--:|
| pyspark-sql / resource / testing / core / errors / logger | 69m34s |
64m26s | 5m08s |
| pyspark-mllib / ml / ml-connect / pipelines | 73m36s | 54m34s | 19m02s |
| pyspark-streaming / structured-streaming / structured-streaming-connect |
57m31s | 41m31s | 16m00s |
| pyspark-connect | 60m26s | 48m01s | 12m25s |
| pyspark-pandas | 71m26s | 56m45s | 14m41s |
| pyspark-pandas-slow | 67m21s | 49m04s | 18m17s |
| pyspark-pandas-connect | 71m57s | 58m54s | 13m03s |
| pyspark-pandas-slow-connect | 72m28s | 59m09s | 13m19s |
| **Sum (8 matrix entries)** | **544m19s** | **432m24s** | **111m55s** |
CI compute totals:
| | Per-run CI time |
|---|---:|
| Sum of 8 matrix entries before | 544m19s |
| Sum of 8 matrix entries after | 432m24s |
| Add the new `precompile` job | +16m14s |
| **Net CI compute saved per run** | **~95m41s (~14% of total ~700m)** |
Wall clock (workflow critical path):
| | Critical path |
|---|---:|
| Before (longest matrix entry: pyspark-mllib/...) | 73m36s |
| After (precompile 16m14s + longest matrix entry: pyspark-sql/... 64m26s)
| ~79m40s |
A roughly +6m wall-clock cost in exchange for ~96m of CI compute savings:
the build was previously parallel-hidden inside each matrix runner; sharing it
serializes one ~16m build before the matrix, but the slowest matrix runner
shrinks by roughly the same amount. Net wall-clock change is small.
### Does this PR introduce _any_ user-facing change?
No. CI infrastructure change only.
### How was this patch tested?
The change is exercised by the CI run of this PR itself:
- If `precompile` succeeds and produces an artifact of reasonable size, the
build phase works.
- If the pyspark matrix completes normally on top of the downloaded
artifact, the artifact is sufficient and `SKIP_SCALA_BUILD` is correctly
skipping the local compile. The "Run tests" step logs `Reusing precompiled
artifact, skipping local SBT build.` to make this visible per matrix entry.
- If the precompile job is forced to fail (or its artifact is missing), the
matrix entries should still pass via the fallback path.
A few things to watch in the first run:
- **Artifact size.** Spark's combined `target/` is roughly 1-3 GB raw;
expect ~1-1.5 GB after gzip. The "Package compile output" step prints the size
with `ls -lh`. If it ever gets close to GHA's 10 GB per-artifact cap we should
slim the find pattern (e.g., exclude `target/streams` and intermediate
scaladoc).
- **`gzip` in the test images.** The pyspark Docker images need `gzip` for
the extract step. It is in the `gzip` package and present in every standard
Ubuntu base image.
The doctests in `dev/sparktestsupport/utils.py` continue to pass; no logic
in `is-changed.py` or the module graph was changed.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Opus 4.7)
Closes #55726 from zhengruifeng/share-sbt-compile-pyspark-sparkr.
Authored-by: Ruifeng Zheng <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
(cherry picked from commit 2c9cd4fb33bd471189a499d66e0cadec45403a58)
Signed-off-by: Hyukjin Kwon <[email protected]>
---
.github/workflows/build_and_test.yml | 92 +++++++++++++++++++++++++++++++++++-
dev/run-tests.py | 6 ++-
2 files changed, 95 insertions(+), 3 deletions(-)
diff --git a/.github/workflows/build_and_test.yml
b/.github/workflows/build_and_test.yml
index 267e54988e75..ffcc49eaec1b 100644
--- a/.github/workflows/build_and_test.yml
+++ b/.github/workflows/build_and_test.yml
@@ -535,8 +535,80 @@ jobs:
cache-from:
type=registry,ref=ghcr.io/apache/spark/apache-spark-github-action-image-pyspark-${{
env.PYSPARK_IMAGE_TO_TEST }}-cache:${{ inputs.branch }}
+ precompile:
+ needs: precondition
+ if: >-
+ (!cancelled()) && (
+ fromJson(needs.precondition.outputs.required).pyspark == 'true' ||
+ fromJson(needs.precondition.outputs.required).pyspark-pandas == 'true'
||
+ fromJson(needs.precondition.outputs.required).pyspark-install ==
'true')
+ name: "Precompile Spark"
+ runs-on: ubuntu-latest
+ timeout-minutes: 60
+ # Optional optimization: if this job fails or is cancelled, the pyspark
+ # matrix entries fall back to running the SBT build locally as before.
+ continue-on-error: true
+ env:
+ HADOOP_PROFILE: ${{ inputs.hadoop }}
+ HIVE_PROFILE: hive2.3
+ GITHUB_PREV_SHA: ${{ github.event.before }}
+ steps:
+ - name: Checkout Spark repository
+ uses: actions/checkout@v6
+ with:
+ fetch-depth: 0
+ repository: apache/spark
+ ref: ${{ inputs.branch }}
+ - name: Sync the current branch with the latest in Apache Spark
+ if: github.repository != 'apache/spark'
+ run: |
+ echo "APACHE_SPARK_REF=$(git rev-parse HEAD)" >> $GITHUB_ENV
+ git fetch https://github.com/$GITHUB_REPOSITORY.git
${GITHUB_REF#refs/heads/}
+ git -c user.name='Apache Spark Test Account' -c
user.email='[email protected]' merge --no-commit --progress --squash
FETCH_HEAD
+ git -c user.name='Apache Spark Test Account' -c
user.email='[email protected]' commit -m "Merged commit" --allow-empty
+ - name: Cache SBT and Maven
+ uses: actions/cache@v5
+ with:
+ path: |
+ build/apache-maven-*
+ build/*.jar
+ ~/.sbt
+ key: build-${{ hashFiles('**/pom.xml', 'project/build.properties',
'build/mvn', 'build/sbt', 'build/sbt-launch-lib.bash',
'build/spark-build-info') }}
+ restore-keys: |
+ build-
+ - name: Cache Coursier local repository
+ uses: actions/cache@v5
+ with:
+ path: ~/.cache/coursier
+ key: precompile-coursier-${{ hashFiles('**/pom.xml', '**/plugins.sbt')
}}
+ restore-keys: |
+ precompile-coursier-
+ - name: Install Java ${{ inputs.java }}
+ uses: actions/setup-java@v5
+ with:
+ distribution: zulu
+ java-version: ${{ inputs.java }}
+ - name: Build Spark
+ run: |
+ ./build/sbt -Phadoop-3 -Pyarn -Pspark-ganglia-lgpl -Phadoop-cloud
-Phive \
+ -Pkubernetes -Pjvm-profiler -Pkinesis-asl -Phive-thriftserver \
+ -Pdocker-integration-tests -Pvolcano \
+ Test/package streaming-kinesis-asl-assembly/assembly
connect/assembly assembly/package
+ - name: Package compile output
+ run: |
+ find . -type d -name target -not -path './build/*' -not -path
'./.git/*' -print0 \
+ | tar --null -czf compile-artifact.tar.gz -T -
+ ls -lh compile-artifact.tar.gz
+ - name: Upload compile artifact
+ uses: actions/upload-artifact@v6
+ with:
+ name: spark-compile-${{ inputs.branch }}-${{ github.run_id }}
+ path: compile-artifact.tar.gz
+ retention-days: 1
+ if-no-files-found: error
+
pyspark:
- needs: [precondition, infra-image]
+ needs: [precondition, infra-image, precompile]
# always run if pyspark == 'true', even infra-image is skip (such as
non-master job)
if: (!cancelled()) &&
(fromJson(needs.precondition.outputs.required).pyspark == 'true' ||
fromJson(needs.precondition.outputs.required).pyspark-pandas == 'true')
name: "Build modules: ${{ matrix.modules }}"
@@ -657,11 +729,29 @@ jobs:
$py -m pip list
echo ""
done
+ - name: Download precompiled artifact
+ id: download-precompiled
+ if: needs.precompile.result == 'success'
+ continue-on-error: true
+ uses: actions/download-artifact@v6
+ with:
+ name: spark-compile-${{ inputs.branch }}-${{ github.run_id }}
+ - name: Extract precompiled artifact
+ id: extract-precompiled
+ if: steps.download-precompiled.outcome == 'success'
+ continue-on-error: true
+ run: |
+ tar -xzf compile-artifact.tar.gz
+ rm compile-artifact.tar.gz
# Run the tests.
- name: Run tests
env: ${{ fromJSON(inputs.envs) }}
shell: 'script -q -e -c "bash {0}"'
run: |
+ if [ "${{ steps.extract-precompiled.outcome }}" = "success" ]; then
+ export SKIP_SCALA_BUILD=true
+ echo "Reusing precompiled artifact, skipping local SBT build."
+ fi
if [[ "$MODULES_TO_TEST" == *"pyspark-pipelines"* ]]; then
export SKIP_PACKAGING=false
echo "Python Packaging Tests Enabled!"
diff --git a/dev/run-tests.py b/dev/run-tests.py
index 0b7a90694385..685621193dd6 100755
--- a/dev/run-tests.py
+++ b/dev/run-tests.py
@@ -644,7 +644,8 @@ def main():
run_build_tests()
# spark build
- build_apache_spark(build_tool, extra_profiles)
+ if os.environ.get("SKIP_SCALA_BUILD", "false") != "true":
+ build_apache_spark(build_tool, extra_profiles)
# backwards compatibility checks
if build_tool == "sbt":
@@ -653,7 +654,8 @@ def main():
detect_binary_inop_with_mima(extra_profiles)
# Since we did not build assembly/package before running dev/mima, we
need to
# do it here because the tests still rely on it; see SPARK-13294 for
details.
- build_spark_assembly_sbt(extra_profiles, should_run_java_style_checks)
+ if os.environ.get("SKIP_SCALA_BUILD", "false") != "true":
+ build_spark_assembly_sbt(extra_profiles,
should_run_java_style_checks)
# run the test suites
run_scala_tests(build_tool, extra_profiles, test_modules, excluded_tags,
included_tags)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
