This is an automated email from the ASF dual-hosted git repository.
gaogaotiantian pushed a commit to branch branch-4.0
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.0 by this push:
new 3583222fba2a [SPARK-57791][4.0][INFRA] Backport CI precompile artifact
sharing, Coursier cache unification, and zstd in CI images
3583222fba2a is described below
commit 3583222fba2a9ad6b73ec423cfbe343e2967804c
Author: Tian Gao <[email protected]>
AuthorDate: Wed Jul 1 15:43:59 2026 -0700
[SPARK-57791][4.0][INFRA] Backport CI precompile artifact sharing, Coursier
cache unification, and zstd in CI images
### What changes were proposed in this pull request?
This backports the CI build-time optimization series to `branch-4.0`,
combining the work that landed on `branch-4.1` as #56798 (precompile artifact
sharing + unified Coursier cache) and #56911 (install `zstd` in the CI
container images) into a single PR.
- **`build_and_test.yml`**: a `precompile` job builds Spark once with SBT
and publishes the compile output (`target/` dirs, zstd-compressed) as an
artifact. The `build`, `pyspark`, `sparkr`, `tpcds-1g`,
`docker-integration-tests`, and `k8s-integration-tests` jobs download and
extract it, falling back to a local SBT build if the precompile job is absent
or fails. The `run-tests`-driven jobs export `SKIP_SCALA_BUILD` to skip the
local build. The per-job Coursier caches are unified under [...]
- **`maven_test.yml`**: a `precompile-maven` job runs `clean install` once
and publishes `target/` plus `~/.m2/.../spark` as an artifact for the matrix
entries to consume.
- **`python_macos_test.yml`**: `branch-4.0`'s analog of `branch-4.1`'s
`python_hosted_runner_test.yml`. A `precompile` job builds Spark with SBT on
macOS and publishes a macOS-distinct artifact that the matrix consumes.
- **`dev/run-tests.py`**: guard `build_apache_spark` and
`build_spark_assembly_sbt` behind `SKIP_SCALA_BUILD` so the test run reuses the
precompiled output.
- **Dockerfiles**: install `zstd` in `dev/infra/Dockerfile` and the `docs`,
`lint`, `sparkr`, `python-311`, and `pypy-310` images under
`dev/spark-test-image/`.
Adaptations for `branch-4.0`:
- `maven_test.yml`: `branch-4.0` has no SPARK-51628 assembly-wipe, so the
full `target/` (including `assembly`) is shipped and the reuse path simply
skips the local `clean install` with no assembly rebuild needed for the connect
tests. No macOS cache guards are added, since SPARK-54466 is not on
`branch-4.0` and the existing build job already caches unconditionally.
- The `zstd` Dockerfile changes cover only the images the `branch-4.0`
scheduler (`branch40_scheduler.yml`) builds: `docs`, `lint`, `sparkr`,
`python-311`, and `pypy-310` (and `dev/infra`). `python-314` does not exist on
`branch-4.0` and is not scheduled, so it is omitted.
### Why are the changes needed?
To cut redundant Scala/Maven compilation and Coursier cache duplication on
`branch-4.0` CI, matching the optimization already present on the newer
branches. Without `zstd` in the container images, `build_and_test.yml`'s `zstd
-dc compile-artifact.tar.zst` extraction fails in the container jobs, and
`actions/cache` silently falls back to `gzip` and never restores caches saved
by host jobs.
### Does this PR introduce _any_ user-facing change?
No. CI-only.
### How was this patch tested?
CI on this PR. The three workflow files validate with `python3 -c "import
yaml; yaml.safe_load(...)"`, and `dev/run-tests.py` compiles cleanly.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (claude-opus-4-8)
Closes #56913 from gaogaotiantian/SPARK-57791-precompile-backport-4.0.
Authored-by: Tian Gao <[email protected]>
Signed-off-by: Tian Gao <[email protected]>
---
.github/workflows/build_and_test.yml | 263 ++++++++++++++++++++++++-----
.github/workflows/maven_test.yml | 101 ++++++++++-
.github/workflows/python_macos_test.yml | 97 ++++++++++-
dev/infra/Dockerfile | 1 +
dev/run-tests.py | 6 +-
dev/spark-test-image/docs/Dockerfile | 1 +
dev/spark-test-image/lint/Dockerfile | 1 +
dev/spark-test-image/pypy-310/Dockerfile | 1 +
dev/spark-test-image/python-311/Dockerfile | 3 +-
dev/spark-test-image/sparkr/Dockerfile | 1 +
10 files changed, 424 insertions(+), 51 deletions(-)
diff --git a/.github/workflows/build_and_test.yml
b/.github/workflows/build_and_test.yml
index cdd76673eb09..05aa487c7b84 100644
--- a/.github/workflows/build_and_test.yml
+++ b/.github/workflows/build_and_test.yml
@@ -223,8 +223,8 @@ jobs:
# Build: build Spark and run the tests for specified modules.
build:
name: "Build modules: ${{ matrix.modules }} ${{ matrix.comment }}"
- needs: precondition
- if: fromJson(needs.precondition.outputs.required).build == 'true'
+ needs: [precondition, precompile]
+ if: (!cancelled()) && fromJson(needs.precondition.outputs.required).build
== 'true'
runs-on: ubuntu-latest
timeout-minutes: 120
strategy:
@@ -331,13 +331,14 @@ jobs:
key: build-${{ hashFiles('**/pom.xml', 'project/build.properties',
'build/mvn', 'build/sbt', 'build/sbt-launch-lib.bash',
'build/spark-build-info') }}
restore-keys: |
build-
- - name: Cache Coursier local repository
- uses: actions/cache@v4
+ - name: Restore Coursier local repository
+ uses: actions/cache/restore@v4
with:
path: ~/.cache/coursier
- key: ${{ matrix.java }}-${{ matrix.hadoop }}-coursier-${{
hashFiles('**/pom.xml', '**/plugins.sbt') }}
+ key: ${{ runner.os }}-${{ matrix.java }}-${{ matrix.hadoop
}}-coursier-${{ hashFiles('**/pom.xml', '**/plugins.sbt') }}
restore-keys: |
- ${{ matrix.java }}-${{ matrix.hadoop }}-coursier-
+ ${{ runner.os }}-${{ matrix.java }}-${{ matrix.hadoop }}-coursier-
+ ${{ runner.os }}-coursier-
- name: Free up disk space
run: |
if [ -f ./dev/free_disk_space ]; then
@@ -362,6 +363,20 @@ jobs:
run: |
python3.11 -m pip install 'numpy>=1.20.0' pyarrow 'pandas==2.3.3'
scipy unittest-xml-reporting 'lxml==4.9.4' 'grpcio==1.67.0'
'grpcio-status==1.67.0' 'protobuf==5.29.1'
python3.11 -m pip list
+ - name: Download precompiled artifact
+ id: download-precompiled
+ if: needs.precompile.result == 'success'
+ continue-on-error: true
+ uses: actions/download-artifact@v4
+ with:
+ name: spark-compile-${{ inputs.branch }}-${{ github.run_id }}
+ - name: Extract precompiled artifact
+ id: extract-precompiled
+ if: steps.download-precompiled.outcome == 'success'
+ continue-on-error: true
+ run: |
+ zstd -dc compile-artifact.tar.zst | tar -xf -
+ rm compile-artifact.tar.zst
# Run the tests.
- name: Run tests
env: ${{ fromJSON(inputs.envs) }}
@@ -372,9 +387,13 @@ jobs:
# Hive "other tests" test needs larger metaspace size based on
experiment.
if [[ "$MODULES_TO_TEST" == "hive" ]] && [[ "$EXCLUDED_TAGS" ==
"org.apache.spark.tags.SlowHiveTest" ]]; then export METASPACE_SIZE=2g; fi
# SPARK-46283: should delete the following env replacement after SPARK
3.x EOL
- if [[ "$MODULES_TO_TEST" == *"streaming-kinesis-asl"* ]] && [[ "${{
inputs.branch }}" =~ ^branch-3 ]]; then
+ if [[ "$MODULES_TO_TEST" == *"streaming-kinesis-asl"* ]] && [[ "${{
inputs.branch }}" =~ ^branch-3 ]]; then
MODULES_TO_TEST=${MODULES_TO_TEST//streaming-kinesis-asl, /}
fi
+ if [ "${{ steps.extract-precompiled.outcome }}" = "success" ]; then
+ export SKIP_SCALA_BUILD=true
+ echo "Reusing precompiled artifact, skipping local SBT build."
+ fi
export SERIAL_SBT_TESTS=1
./dev/run-tests --parallelism 1 --modules "$MODULES_TO_TEST"
--included-tags "$INCLUDED_TAGS" --excluded-tags "$EXCLUDED_TAGS"
- name: Upload test results to report
@@ -485,8 +504,84 @@ jobs:
cache-from:
type=registry,ref=ghcr.io/apache/spark/apache-spark-github-action-image-pyspark-${{
env.PYSPARK_IMAGE_TO_TEST }}-cache:${{ inputs.branch }}
+ precompile:
+ needs: precondition
+ if: >-
+ (!cancelled()) && (
+ fromJson(needs.precondition.outputs.required).build == 'true' ||
+ fromJson(needs.precondition.outputs.required).pyspark == 'true' ||
+ fromJson(needs.precondition.outputs.required).pyspark-pandas == 'true'
||
+ fromJson(needs.precondition.outputs.required).sparkr == 'true' ||
+ fromJson(needs.precondition.outputs.required).docker-integration-tests
== 'true' ||
+ fromJson(needs.precondition.outputs.required).k8s-integration-tests ==
'true' ||
+ fromJson(needs.precondition.outputs.required).tpcds-1g == 'true')
+ name: "Precompile Spark"
+ runs-on: ubuntu-latest
+ timeout-minutes: 60
+ # Optional optimization: if this job fails or is cancelled, the pyspark
+ # matrix entries fall back to running the SBT build locally as before.
+ continue-on-error: true
+ env:
+ HADOOP_PROFILE: ${{ inputs.hadoop }}
+ HIVE_PROFILE: hive2.3
+ GITHUB_PREV_SHA: ${{ github.event.before }}
+ steps:
+ - name: Checkout Spark repository
+ uses: actions/checkout@v4
+ with:
+ fetch-depth: 0
+ repository: apache/spark
+ ref: ${{ inputs.branch }}
+ - name: Sync the current branch with the latest in Apache Spark
+ if: github.repository != 'apache/spark'
+ run: |
+ echo "APACHE_SPARK_REF=$(git rev-parse HEAD)" >> $GITHUB_ENV
+ git fetch https://github.com/$GITHUB_REPOSITORY.git
${GITHUB_REF#refs/heads/}
+ git -c user.name='Apache Spark Test Account' -c
user.email='[email protected]' merge --no-commit --progress --squash
FETCH_HEAD
+ git -c user.name='Apache Spark Test Account' -c
user.email='[email protected]' commit -m "Merged commit" --allow-empty
+ - name: Cache SBT and Maven
+ uses: actions/cache@v4
+ with:
+ path: |
+ build/apache-maven-*
+ build/*.jar
+ ~/.sbt
+ key: build-${{ hashFiles('**/pom.xml', 'project/build.properties',
'build/mvn', 'build/sbt', 'build/sbt-launch-lib.bash',
'build/spark-build-info') }}
+ restore-keys: |
+ build-
+ - name: Cache Coursier local repository
+ uses: actions/cache@v4
+ with:
+ path: ~/.cache/coursier
+ key: ${{ runner.os }}-coursier-${{ hashFiles('**/pom.xml',
'**/plugins.sbt') }}
+ restore-keys: |
+ ${{ runner.os }}-coursier-
+ - name: Install Java ${{ inputs.java }}
+ uses: actions/setup-java@v4
+ with:
+ distribution: zulu
+ java-version: ${{ inputs.java }}
+ - name: Build Spark
+ run: |
+ ./build/sbt -Phadoop-3 -Pyarn -Pspark-ganglia-lgpl -Phadoop-cloud
-Phive \
+ -Pkubernetes -Pjvm-profiler -Pkinesis-asl -Phive-thriftserver \
+ -Pdocker-integration-tests -Pkubernetes-integration-tests -Pvolcano \
+ Test/package streaming-kinesis-asl-assembly/assembly
connect/assembly assembly/package
+ - name: Package compile output
+ run: |
+ find . -type d -name target -not -path './build/*' -not -path
'./.git/*' -print0 \
+ | tar --null -cf - -T - | zstd -c -T0 > compile-artifact.tar.zst
+ ls -lh compile-artifact.tar.zst
+ - name: Upload compile artifact
+ uses: actions/upload-artifact@v4
+ with:
+ name: spark-compile-${{ inputs.branch }}-${{ github.run_id }}
+ path: compile-artifact.tar.zst
+ retention-days: 1
+ if-no-files-found: error
+
pyspark:
- needs: [precondition, infra-image]
+ needs: [precondition, infra-image, precompile]
# always run if pyspark == 'true', even infra-image is skip (such as
non-master job)
if: (!cancelled()) &&
(fromJson(needs.precondition.outputs.required).pyspark == 'true' ||
fromJson(needs.precondition.outputs.required).pyspark-pandas == 'true')
name: "Build modules: ${{ matrix.modules }}"
@@ -573,13 +668,13 @@ jobs:
key: build-${{ hashFiles('**/pom.xml', 'project/build.properties',
'build/mvn', 'build/sbt', 'build/sbt-launch-lib.bash',
'build/spark-build-info') }}
restore-keys: |
build-
- - name: Cache Coursier local repository
- uses: actions/cache@v4
+ - name: Restore Coursier local repository
+ uses: actions/cache/restore@v4
with:
path: ~/.cache/coursier
- key: pyspark-coursier-${{ hashFiles('**/pom.xml', '**/plugins.sbt') }}
+ key: ${{ runner.os }}-coursier-${{ hashFiles('**/pom.xml',
'**/plugins.sbt') }}
restore-keys: |
- pyspark-coursier-
+ ${{ runner.os }}-coursier-
- name: Free up disk space
shell: 'script -q -e -c "bash {0}"'
run: ./dev/free_disk_space_container
@@ -603,11 +698,29 @@ jobs:
uses: conda-incubator/setup-miniconda@v3
with:
miniforge-version: latest
+ - name: Download precompiled artifact
+ id: download-precompiled
+ if: needs.precompile.result == 'success'
+ continue-on-error: true
+ uses: actions/download-artifact@v4
+ with:
+ name: spark-compile-${{ inputs.branch }}-${{ github.run_id }}
+ - name: Extract precompiled artifact
+ id: extract-precompiled
+ if: steps.download-precompiled.outcome == 'success'
+ continue-on-error: true
+ run: |
+ zstd -dc compile-artifact.tar.zst | tar -xf -
+ rm compile-artifact.tar.zst
# Run the tests.
- name: Run tests
env: ${{ fromJSON(inputs.envs) }}
shell: 'script -q -e -c "bash {0}"'
run: |
+ if [ "${{ steps.extract-precompiled.outcome }}" = "success" ]; then
+ export SKIP_SCALA_BUILD=true
+ echo "Reusing precompiled artifact, skipping local SBT build."
+ fi
if [[ "$MODULES_TO_TEST" == *"pyspark-errors"* ]]; then
export PATH=$CONDA/bin:$PATH
export SKIP_PACKAGING=false
@@ -645,7 +758,7 @@ jobs:
path: "**/target/unit-tests.log"
sparkr:
- needs: [precondition, infra-image]
+ needs: [precondition, infra-image, precompile]
# always run if sparkr == 'true', even infra-image is skip (such as
non-master job)
if: (!cancelled()) && fromJson(needs.precondition.outputs.required).sparkr
== 'true'
name: "Build modules: sparkr"
@@ -690,13 +803,13 @@ jobs:
key: build-${{ hashFiles('**/pom.xml', 'project/build.properties',
'build/mvn', 'build/sbt', 'build/sbt-launch-lib.bash',
'build/spark-build-info') }}
restore-keys: |
build-
- - name: Cache Coursier local repository
- uses: actions/cache@v4
+ - name: Restore Coursier local repository
+ uses: actions/cache/restore@v4
with:
path: ~/.cache/coursier
- key: sparkr-coursier-${{ hashFiles('**/pom.xml', '**/plugins.sbt') }}
+ key: ${{ runner.os }}-coursier-${{ hashFiles('**/pom.xml',
'**/plugins.sbt') }}
restore-keys: |
- sparkr-coursier-
+ ${{ runner.os }}-coursier-
- name: Free up disk space
run: ./dev/free_disk_space_container
- name: Install Java ${{ inputs.java }}
@@ -704,6 +817,20 @@ jobs:
with:
distribution: zulu
java-version: ${{ inputs.java }}
+ - name: Download precompiled artifact
+ id: download-precompiled
+ if: needs.precompile.result == 'success'
+ continue-on-error: true
+ uses: actions/download-artifact@v4
+ with:
+ name: spark-compile-${{ inputs.branch }}-${{ github.run_id }}
+ - name: Extract precompiled artifact
+ id: extract-precompiled
+ if: steps.download-precompiled.outcome == 'success'
+ continue-on-error: true
+ run: |
+ zstd -dc compile-artifact.tar.zst | tar -xf -
+ rm compile-artifact.tar.zst
- name: Run tests
env: ${{ fromJSON(inputs.envs) }}
run: |
@@ -711,6 +838,10 @@ jobs:
# R issues at docker environment
export TZ=UTC
export _R_CHECK_SYSTEM_CLOCK_=FALSE
+ if [ "${{ steps.extract-precompiled.outcome }}" = "success" ]; then
+ export SKIP_SCALA_BUILD=true
+ echo "Reusing precompiled artifact, skipping local SBT build."
+ fi
./dev/run-tests --parallelism 1 --modules sparkr
- name: Upload test results to report
if: always()
@@ -811,13 +942,13 @@ jobs:
key: build-${{ hashFiles('**/pom.xml', 'project/build.properties',
'build/mvn', 'build/sbt', 'build/sbt-launch-lib.bash',
'build/spark-build-info') }}
restore-keys: |
build-
- - name: Cache Coursier local repository
- uses: actions/cache@v4
+ - name: Restore Coursier local repository
+ uses: actions/cache/restore@v4
with:
path: ~/.cache/coursier
- key: docs-coursier-${{ hashFiles('**/pom.xml', '**/plugins.sbt') }}
+ key: ${{ runner.os }}-coursier-${{ hashFiles('**/pom.xml',
'**/plugins.sbt') }}
restore-keys: |
- docs-coursier-
+ ${{ runner.os }}-coursier-
- name: Cache Maven local repository
uses: actions/cache@v4
with:
@@ -942,13 +1073,13 @@ jobs:
key: build-${{ hashFiles('**/pom.xml', 'project/build.properties',
'build/mvn', 'build/sbt', 'build/sbt-launch-lib.bash',
'build/spark-build-info') }}
restore-keys: |
build-
- - name: Cache Coursier local repository
- uses: actions/cache@v4
+ - name: Restore Coursier local repository
+ uses: actions/cache/restore@v4
with:
path: ~/.cache/coursier
- key: docs-coursier-${{ hashFiles('**/pom.xml', '**/plugins.sbt') }}
+ key: ${{ runner.os }}-coursier-${{ hashFiles('**/pom.xml',
'**/plugins.sbt') }}
restore-keys: |
- docs-coursier-
+ ${{ runner.os }}-coursier-
- name: Cache Maven local repository
uses: actions/cache@v4
with:
@@ -1022,8 +1153,8 @@ jobs:
# Any TPC-DS related updates on this job need to be applied to tpcds-1g-gen
job of benchmark.yml as well
tpcds-1g:
- needs: precondition
- if: fromJson(needs.precondition.outputs.required).tpcds-1g == 'true'
+ needs: [precondition, precompile]
+ if: (!cancelled()) &&
fromJson(needs.precondition.outputs.required).tpcds-1g == 'true'
name: Run TPC-DS queries with SF=1
runs-on: ubuntu-latest
timeout-minutes: 120
@@ -1052,18 +1183,32 @@ jobs:
key: build-${{ hashFiles('**/pom.xml', 'project/build.properties',
'build/mvn', 'build/sbt', 'build/sbt-launch-lib.bash',
'build/spark-build-info') }}
restore-keys: |
build-
- - name: Cache Coursier local repository
- uses: actions/cache@v4
+ - name: Restore Coursier local repository
+ uses: actions/cache/restore@v4
with:
path: ~/.cache/coursier
- key: tpcds-coursier-${{ hashFiles('**/pom.xml', '**/plugins.sbt') }}
+ key: ${{ runner.os }}-coursier-${{ hashFiles('**/pom.xml',
'**/plugins.sbt') }}
restore-keys: |
- tpcds-coursier-
+ ${{ runner.os }}-coursier-
- name: Install Java ${{ inputs.java }}
uses: actions/setup-java@v4
with:
distribution: zulu
java-version: ${{ inputs.java }}
+ - name: Download precompiled artifact
+ id: download-precompiled
+ if: needs.precompile.result == 'success'
+ continue-on-error: true
+ uses: actions/download-artifact@v4
+ with:
+ name: spark-compile-${{ inputs.branch }}-${{ github.run_id }}
+ - name: Extract precompiled artifact
+ id: extract-precompiled
+ if: steps.download-precompiled.outcome == 'success'
+ continue-on-error: true
+ run: |
+ zstd -dc compile-artifact.tar.zst | tar -xf -
+ rm compile-artifact.tar.zst
- name: Cache TPC-DS generated data
id: cache-tpcds-sf-1
uses: actions/cache@v4
@@ -1124,8 +1269,8 @@ jobs:
path: "**/target/unit-tests.log"
docker-integration-tests:
- needs: precondition
- if: fromJson(needs.precondition.outputs.required).docker-integration-tests
== 'true'
+ needs: [precondition, precompile]
+ if: (!cancelled()) &&
fromJson(needs.precondition.outputs.required).docker-integration-tests == 'true'
name: Run Docker integration tests
runs-on: ubuntu-latest
timeout-minutes: 120
@@ -1161,21 +1306,39 @@ jobs:
key: build-${{ hashFiles('**/pom.xml', 'project/build.properties',
'build/mvn', 'build/sbt', 'build/sbt-launch-lib.bash',
'build/spark-build-info') }}
restore-keys: |
build-
- - name: Cache Coursier local repository
- uses: actions/cache@v4
+ - name: Restore Coursier local repository
+ uses: actions/cache/restore@v4
with:
path: ~/.cache/coursier
- key: docker-integration-coursier-${{ hashFiles('**/pom.xml',
'**/plugins.sbt') }}
+ key: ${{ runner.os }}-coursier-${{ hashFiles('**/pom.xml',
'**/plugins.sbt') }}
restore-keys: |
- docker-integration-coursier-
+ ${{ runner.os }}-coursier-
- name: Install Java ${{ inputs.java }}
uses: actions/setup-java@v4
with:
distribution: zulu
java-version: ${{ inputs.java }}
+ - name: Download precompiled artifact
+ id: download-precompiled
+ if: needs.precompile.result == 'success'
+ continue-on-error: true
+ uses: actions/download-artifact@v4
+ with:
+ name: spark-compile-${{ inputs.branch }}-${{ github.run_id }}
+ - name: Extract precompiled artifact
+ id: extract-precompiled
+ if: steps.download-precompiled.outcome == 'success'
+ continue-on-error: true
+ run: |
+ zstd -dc compile-artifact.tar.zst | tar -xf -
+ rm compile-artifact.tar.zst
- name: Run tests
env: ${{ fromJSON(inputs.envs) }}
run: |
+ if [ "${{ steps.extract-precompiled.outcome }}" = "success" ]; then
+ export SKIP_SCALA_BUILD=true
+ echo "Reusing precompiled artifact, skipping local SBT build."
+ fi
./dev/run-tests --parallelism 1 --modules docker-integration-tests
--included-tags org.apache.spark.tags.DockerTest
- name: Upload test results to report
if: always()
@@ -1191,8 +1354,8 @@ jobs:
path: "**/target/unit-tests.log"
k8s-integration-tests:
- needs: precondition
- if: fromJson(needs.precondition.outputs.required).k8s-integration-tests ==
'true'
+ needs: [precondition, precompile]
+ if: (!cancelled()) &&
fromJson(needs.precondition.outputs.required).k8s-integration-tests == 'true'
name: Run Spark on Kubernetes Integration test
runs-on: ubuntu-latest
timeout-minutes: 120
@@ -1220,13 +1383,13 @@ jobs:
key: build-${{ hashFiles('**/pom.xml', 'project/build.properties',
'build/mvn', 'build/sbt', 'build/sbt-launch-lib.bash',
'build/spark-build-info') }}
restore-keys: |
build-
- - name: Cache Coursier local repository
- uses: actions/cache@v4
+ - name: Restore Coursier local repository
+ uses: actions/cache/restore@v4
with:
path: ~/.cache/coursier
- key: k8s-integration-coursier-${{ hashFiles('**/pom.xml',
'**/plugins.sbt') }}
+ key: ${{ runner.os }}-coursier-${{ hashFiles('**/pom.xml',
'**/plugins.sbt') }}
restore-keys: |
- k8s-integration-coursier-
+ ${{ runner.os }}-coursier-
- name: Free up disk space
run: |
if [ -f ./dev/free_disk_space ]; then
@@ -1237,6 +1400,20 @@ jobs:
with:
distribution: zulu
java-version: ${{ inputs.java }}
+ - name: Download precompiled artifact
+ id: download-precompiled
+ if: needs.precompile.result == 'success'
+ continue-on-error: true
+ uses: actions/download-artifact@v4
+ with:
+ name: spark-compile-${{ inputs.branch }}-${{ github.run_id }}
+ - name: Extract precompiled artifact
+ id: extract-precompiled
+ if: steps.download-precompiled.outcome == 'success'
+ continue-on-error: true
+ run: |
+ zstd -dc compile-artifact.tar.zst | tar -xf -
+ rm compile-artifact.tar.zst
- name: Install R
run: |
sudo apt update
diff --git a/.github/workflows/maven_test.yml b/.github/workflows/maven_test.yml
index 67813815d43b..1fabe6c3e9cb 100644
--- a/.github/workflows/maven_test.yml
+++ b/.github/workflows/maven_test.yml
@@ -47,9 +47,85 @@ on:
type: string
default: '{}'
jobs:
+ # Precompile Spark with Maven once and publish target/ + ~/.m2/.../spark as
+ # an artifact for the matrix entries below to consume. Optional: any failure
+ # here degrades the matrix to its original local `clean install` path.
+ precompile-maven:
+ name: "Precompile Spark with Maven"
+ runs-on: ${{ inputs.os }}
+ # If this job fails or is cancelled, the matrix entries fall back to
+ # running `mvn clean install` locally as before.
+ continue-on-error: true
+ env:
+ HADOOP_PROFILE: ${{ inputs.hadoop }}
+ HIVE_PROFILE: hive2.3
+ SPARK_LOCAL_IP: localhost
+ GITHUB_PREV_SHA: ${{ github.event.before }}
+ steps:
+ - name: Checkout Spark repository
+ uses: actions/checkout@v4
+ with:
+ fetch-depth: 0
+ repository: apache/spark
+ ref: ${{ inputs.branch }}
+ - name: Sync the current branch with the latest in Apache Spark
+ if: github.repository != 'apache/spark'
+ run: |
+ echo "APACHE_SPARK_REF=$(git rev-parse HEAD)" >> $GITHUB_ENV
+ git fetch https://github.com/$GITHUB_REPOSITORY.git
${GITHUB_REF#refs/heads/}
+ git -c user.name='Apache Spark Test Account' -c
user.email='[email protected]' merge --no-commit --progress --squash
FETCH_HEAD
+ git -c user.name='Apache Spark Test Account' -c
user.email='[email protected]' commit -m "Merged commit" --allow-empty
+ - name: Cache SBT and Maven
+ uses: actions/cache@v4
+ with:
+ path: |
+ build/apache-maven-*
+ build/*.jar
+ ~/.sbt
+ key: build-${{ hashFiles('**/pom.xml', 'project/build.properties',
'build/mvn', 'build/sbt', 'build/sbt-launch-lib.bash',
'build/spark-build-info') }}
+ restore-keys: |
+ build-
+ - name: Cache Maven local repository
+ uses: actions/cache@v4
+ with:
+ path: ~/.m2/repository
+ key: java${{ inputs.java }}-maven-${{ hashFiles('**/pom.xml') }}
+ restore-keys: |
+ java${{ inputs.java }}-maven-
+ - name: Install Java ${{ inputs.java }}
+ uses: actions/setup-java@v4
+ with:
+ distribution: zulu
+ java-version: ${{ inputs.java }}
+ - name: Build Spark with Maven
+ run: |
+ export MAVEN_OPTS="-Xss64m -Xmx4g -Xms4g
-XX:ReservedCodeCacheSize=128m -Dorg.slf4j.simpleLogger.defaultLogLevel=WARN"
+ export MAVEN_CLI_OPTS="--no-transfer-progress"
+ export JAVA_VERSION=${{ inputs.java }}
+ ./build/mvn $MAVEN_CLI_OPTS -DskipTests -Pyarn -Pkubernetes
-Pvolcano -Phive -Phive-thriftserver -Phadoop-cloud -Pjvm-profiler
-Pspark-ganglia-lgpl -Pkinesis-asl -Djava.version=${JAVA_VERSION/-ea} clean
install
+ - name: Package compile output
+ run: |
+ find . -type d -name target -not -path './build/*' -not -path
'./.git/*' -print0 \
+ | tar --null -cf - -T - | zstd -c -T0 > compile-target.tar.zst
+ if [ -d "$HOME/.m2/repository/org/apache/spark" ]; then
+ tar -C "$HOME/.m2/repository/org/apache" -cf - spark | zstd -c -T0
> compile-m2-spark.tar.zst
+ fi
+ ls -lh compile-target.tar.zst compile-m2-spark.tar.zst
+ - name: Upload compile artifact
+ uses: actions/upload-artifact@v4
+ with:
+ name: spark-maven-compile-${{ inputs.branch }}-java${{ inputs.java
}}-${{ github.run_id }}
+ path: |
+ compile-target.tar.zst
+ compile-m2-spark.tar.zst
+ retention-days: 1
+ if-no-files-found: error
+
# Build: build Spark and run the tests for specified modules using maven
build:
name: "Build modules using Maven: ${{ matrix.modules }} ${{ matrix.comment
}}"
+ needs: precompile-maven
+ if: (!cancelled())
runs-on: ${{ inputs.os }}
timeout-minutes: 150
strategy:
@@ -182,6 +258,25 @@ jobs:
run: |
python3.11 -m pip install 'numpy>=1.20.0' pyarrow 'pandas==2.3.3'
scipy unittest-xml-reporting 'grpcio==1.67.0' 'grpcio-status==1.67.0'
'protobuf==5.29.1'
python3.11 -m pip list
+ - name: Download precompiled artifact
+ id: download-precompiled
+ if: needs.precompile-maven.result == 'success'
+ continue-on-error: true
+ uses: actions/download-artifact@v4
+ with:
+ name: spark-maven-compile-${{ inputs.branch }}-java${{ matrix.java
}}-${{ github.run_id }}
+ - name: Extract precompiled artifact
+ id: extract-precompiled
+ if: steps.download-precompiled.outcome == 'success'
+ continue-on-error: true
+ run: |
+ zstd -dc compile-target.tar.zst | tar -xf -
+ rm compile-target.tar.zst
+ if [ -f compile-m2-spark.tar.zst ]; then
+ mkdir -p "$HOME/.m2/repository/org/apache"
+ zstd -dc compile-m2-spark.tar.zst | tar -C
"$HOME/.m2/repository/org/apache" -xf -
+ rm compile-m2-spark.tar.zst
+ fi
# Run the tests.
- name: Run tests
env: ${{ fromJSON(inputs.envs) }}
@@ -192,7 +287,11 @@ jobs:
export ENABLE_KINESIS_TESTS=0
# Replace with the real module name, for example,
connector#kafka-0-10 -> connector/kafka-0-10
export TEST_MODULES=`echo "$MODULES_TO_TEST" | sed -e "s%#%/%g"`
- ./build/mvn $MAVEN_CLI_OPTS -DskipTests -Pyarn -Pkubernetes
-Pvolcano -Phive -Phive-thriftserver -Phadoop-cloud -Pjvm-profiler
-Pspark-ganglia-lgpl -Pkinesis-asl -Djava.version=${JAVA_VERSION/-ea} clean
install
+ if [ "${{ steps.extract-precompiled.outcome }}" = "success" ]; then
+ echo "Reusing precompiled artifact, skipping local Maven clean
install."
+ else
+ ./build/mvn $MAVEN_CLI_OPTS -DskipTests -Pyarn -Pkubernetes
-Pvolcano -Phive -Phive-thriftserver -Phadoop-cloud -Pjvm-profiler
-Pspark-ganglia-lgpl -Pkinesis-asl -Djava.version=${JAVA_VERSION/-ea} clean
install
+ fi
if [[ "$INCLUDED_TAGS" != "" ]]; then
./build/mvn $MAVEN_CLI_OPTS -pl "$TEST_MODULES" -Pyarn
-Pkubernetes -Pvolcano -Phive -Phive-thriftserver -Phadoop-cloud -Pjvm-profiler
-Pspark-ganglia-lgpl -Pkinesis-asl -Djava.version=${JAVA_VERSION/-ea}
-Dtest.include.tags="$INCLUDED_TAGS" test -fae
elif [[ "$MODULES_TO_TEST" == "connect" ]]; then
diff --git a/.github/workflows/python_macos_test.yml
b/.github/workflows/python_macos_test.yml
index bce464fe0826..de682ee36e53 100644
--- a/.github/workflows/python_macos_test.yml
+++ b/.github/workflows/python_macos_test.yml
@@ -46,8 +46,79 @@ on:
type: string
default: '{}'
jobs:
+ # Precompile Spark with SBT once and publish target/ as an artifact for the
+ # matrix entries below to consume. Optional: any failure here degrades the
+ # matrix to its original local SBT build path.
+ precompile:
+ name: "Precompile Spark"
+ runs-on: macos-15
+ # If this job fails or is cancelled, the matrix entries fall back to
+ # running the SBT build locally as before.
+ continue-on-error: true
+ env:
+ HADOOP_PROFILE: ${{ inputs.hadoop }}
+ HIVE_PROFILE: hive2.3
+ SPARK_LOCAL_IP: localhost
+ GITHUB_PREV_SHA: ${{ github.event.before }}
+ steps:
+ - name: Checkout Spark repository
+ uses: actions/checkout@v4
+ with:
+ fetch-depth: 0
+ repository: apache/spark
+ ref: ${{ inputs.branch }}
+ - name: Sync the current branch with the latest in Apache Spark
+ if: github.repository != 'apache/spark'
+ run: |
+ echo "APACHE_SPARK_REF=$(git rev-parse HEAD)" >> $GITHUB_ENV
+ git fetch https://github.com/$GITHUB_REPOSITORY.git
${GITHUB_REF#refs/heads/}
+ git -c user.name='Apache Spark Test Account' -c
user.email='[email protected]' merge --no-commit --progress --squash
FETCH_HEAD
+ git -c user.name='Apache Spark Test Account' -c
user.email='[email protected]' commit -m "Merged commit" --allow-empty
+ - name: Cache SBT and Maven
+ uses: actions/cache@v4
+ with:
+ path: |
+ build/apache-maven-*
+ build/*.jar
+ ~/.sbt
+ key: build-${{ hashFiles('**/pom.xml', 'project/build.properties',
'build/mvn', 'build/sbt', 'build/sbt-launch-lib.bash',
'build/spark-build-info') }}
+ restore-keys: |
+ build-
+ - name: Cache Coursier local repository
+ uses: actions/cache@v4
+ with:
+ path: ~/.cache/coursier
+ key: ${{ runner.os }}-coursier-${{ hashFiles('**/pom.xml',
'**/plugins.sbt') }}
+ restore-keys: |
+ ${{ runner.os }}-coursier-
+ - name: Install Java ${{ inputs.java }}
+ uses: actions/setup-java@v4
+ with:
+ distribution: zulu
+ java-version: ${{ inputs.java }}
+ - name: Build Spark
+ run: |
+ ./build/sbt -Phadoop-3 -Pyarn -Pspark-ganglia-lgpl -Phadoop-cloud
-Phive \
+ -Pkubernetes -Pjvm-profiler -Pkinesis-asl -Phive-thriftserver \
+ -Pdocker-integration-tests -Pvolcano \
+ Test/package streaming-kinesis-asl-assembly/assembly
connect/assembly assembly/package
+ - name: Package compile output
+ run: |
+ find . -type d -name target -not -path './build/*' -not -path
'./.git/*' -print0 \
+ | tar --null -cf - -T - | zstd -c -T0 > compile-artifact.tar.zst
+ ls -lh compile-artifact.tar.zst
+ - name: Upload compile artifact
+ uses: actions/upload-artifact@v4
+ with:
+ name: spark-compile-macos-${{ inputs.branch }}-${{ github.run_id }}
+ path: compile-artifact.tar.zst
+ retention-days: 1
+ if-no-files-found: error
+
build:
name: "PySpark test on macos: ${{ matrix.modules }}"
+ needs: precompile
+ if: (!cancelled())
runs-on: macos-15
strategy:
fail-fast: false
@@ -114,13 +185,13 @@ jobs:
key: build-${{ hashFiles('**/pom.xml', 'project/build.properties',
'build/mvn', 'build/sbt', 'build/sbt-launch-lib.bash',
'build/spark-build-info') }}
restore-keys: |
build-
- - name: Cache Coursier local repository
- uses: actions/cache@v4
+ - name: Restore Coursier local repository
+ uses: actions/cache/restore@v4
with:
path: ~/.cache/coursier
- key: pyspark-coursier-${{ hashFiles('**/pom.xml', '**/plugins.sbt')
}}
+ key: ${{ runner.os }}-coursier-${{ hashFiles('**/pom.xml',
'**/plugins.sbt') }}
restore-keys: |
- pyspark-coursier-
+ ${{ runner.os }}-coursier-
- name: Install Java ${{ matrix.java }}
uses: actions/setup-java@v4
with:
@@ -135,10 +206,28 @@ jobs:
python${{matrix.python}} -m pip cache purge
- name: List Python packages
run: python${{matrix.python}} -m pip list
+ - name: Download precompiled artifact
+ id: download-precompiled
+ if: needs.precompile.result == 'success'
+ continue-on-error: true
+ uses: actions/download-artifact@v4
+ with:
+ name: spark-compile-macos-${{ inputs.branch }}-${{ github.run_id }}
+ - name: Extract precompiled artifact
+ id: extract-precompiled
+ if: steps.download-precompiled.outcome == 'success'
+ continue-on-error: true
+ run: |
+ zstd -dc compile-artifact.tar.zst | tar -xf -
+ rm compile-artifact.tar.zst
# Run the tests.
- name: Run tests
env: ${{ fromJSON(inputs.envs) }}
run: |
+ if [ "${{ steps.extract-precompiled.outcome }}" = "success" ]; then
+ export SKIP_SCALA_BUILD=true
+ echo "Reusing precompiled artifact, skipping local SBT build."
+ fi
if [[ "$MODULES_TO_TEST" == *"pyspark-errors"* ]]; then
export SKIP_PACKAGING=false
echo "Python Packaging Tests Enabled!"
diff --git a/dev/infra/Dockerfile b/dev/infra/Dockerfile
index 8965802fcb75..548389c2c437 100644
--- a/dev/infra/Dockerfile
+++ b/dev/infra/Dockerfile
@@ -66,6 +66,7 @@ RUN apt-get update && apt-get install -y \
software-properties-common \
wget \
zlib1g-dev \
+ zstd \
&& rm -rf /var/lib/apt/lists/*
diff --git a/dev/run-tests.py b/dev/run-tests.py
index eb760139f9b6..5425ccec2072 100755
--- a/dev/run-tests.py
+++ b/dev/run-tests.py
@@ -642,7 +642,8 @@ def main():
run_build_tests()
# spark build
- build_apache_spark(build_tool, extra_profiles)
+ if os.environ.get("SKIP_SCALA_BUILD", "false") != "true":
+ build_apache_spark(build_tool, extra_profiles)
# backwards compatibility checks
if build_tool == "sbt":
@@ -651,7 +652,8 @@ def main():
detect_binary_inop_with_mima(extra_profiles)
# Since we did not build assembly/package before running dev/mima, we
need to
# do it here because the tests still rely on it; see SPARK-13294 for
details.
- build_spark_assembly_sbt(extra_profiles, should_run_java_style_checks)
+ if os.environ.get("SKIP_SCALA_BUILD", "false") != "true":
+ build_spark_assembly_sbt(extra_profiles,
should_run_java_style_checks)
# run the test suites
run_scala_tests(build_tool, extra_profiles, test_modules, excluded_tags,
included_tags)
diff --git a/dev/spark-test-image/docs/Dockerfile
b/dev/spark-test-image/docs/Dockerfile
index 6735795adef4..4e35a36f5095 100644
--- a/dev/spark-test-image/docs/Dockerfile
+++ b/dev/spark-test-image/docs/Dockerfile
@@ -65,6 +65,7 @@ RUN apt-get update && apt-get install -y \
software-properties-common \
wget \
zlib1g-dev \
+ zstd \
&& rm -rf /var/lib/apt/lists/*
diff --git a/dev/spark-test-image/lint/Dockerfile
b/dev/spark-test-image/lint/Dockerfile
index 5c2157fc7258..e25fdde691b5 100644
--- a/dev/spark-test-image/lint/Dockerfile
+++ b/dev/spark-test-image/lint/Dockerfile
@@ -58,6 +58,7 @@ RUN apt-get update && apt-get install -y \
software-properties-common \
wget \
zlib1g-dev \
+ zstd \
&& rm -rf /var/lib/apt/lists/*
RUN Rscript -e "install.packages(c('remotes', 'knitr', 'markdown',
'rmarkdown', 'testthat'), repos='https://cloud.r-project.org/')" \
diff --git a/dev/spark-test-image/pypy-310/Dockerfile
b/dev/spark-test-image/pypy-310/Dockerfile
index 8461ed5bd4c9..2a9f6538d50f 100644
--- a/dev/spark-test-image/pypy-310/Dockerfile
+++ b/dev/spark-test-image/pypy-310/Dockerfile
@@ -58,6 +58,7 @@ RUN apt-get update && apt-get install -y \
software-properties-common \
wget \
zlib1g-dev \
+ zstd \
&& apt-get autoremove --purge -y \
&& apt-get clean \
&& rm -rf /var/lib/apt/lists/*
diff --git a/dev/spark-test-image/python-311/Dockerfile
b/dev/spark-test-image/python-311/Dockerfile
index 047e91077b71..257ce34bcddc 100644
--- a/dev/spark-test-image/python-311/Dockerfile
+++ b/dev/spark-test-image/python-311/Dockerfile
@@ -57,7 +57,8 @@ RUN apt-get update && apt-get install -y \
tzdata \
software-properties-common \
wget \
- zlib1g-dev
+ zlib1g-dev \
+ zstd
# Install Python 3.11
RUN add-apt-repository ppa:deadsnakes/ppa
diff --git a/dev/spark-test-image/sparkr/Dockerfile
b/dev/spark-test-image/sparkr/Dockerfile
index 9b2d702fc2d6..26554d693a9c 100644
--- a/dev/spark-test-image/sparkr/Dockerfile
+++ b/dev/spark-test-image/sparkr/Dockerfile
@@ -60,6 +60,7 @@ RUN apt-get update && apt-get install -y \
software-properties-common \
wget \
zlib1g-dev \
+ zstd \
&& rm -rf /var/lib/apt/lists/*
RUN echo 'deb https://cloud.r-project.org/bin/linux/ubuntu jammy-cran40/' >>
/etc/apt/sources.list
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
