This is an automated email from the ASF dual-hosted git repository.
nchammas pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new e0761c3844d9 [SPARK-59086][SPARK-54466][BUILD] Extract common build
env setup into composite action
e0761c3844d9 is described below
commit e0761c3844d9ec1e651a7168b04582e4df54cad5
Author: Nicholas Chammas <[email protected]>
AuthorDate: Wed Sep 30 11:24:32 2026 +0300
[SPARK-59086][SPARK-54466][BUILD] Extract common build env setup into
composite action
### What changes were proposed in this pull request?
1. Extract the repeated build environment setup across several workflows
into a common composite action.
2. Collapse the various cache keys used into a small set in the composite
action.
3. Remove the various skips and comments related to macOS.
Related: #57187
### Why are the changes needed?
1. We should capture repeated workflow steps in a composite action to make
them easier to maintain over time and to prevent unintentional drift across the
various copies of the steps.
2. I intentionally collapsed the various cache keys because I didn't see
that they were serving a purpose. If we find over time that there are cache
misses costing us build time, we can update the composite action and, if
necessary, parametrize some of the cache keys. Until then, however, I erred on
the side of simplicity.
For example, the `build_python_connect*.yml` workflows intentionally use
restore-only Coursier caching. They consume the shared Linux cache written by
`precompile` jobs rather than creating their own potentially thin cache entry.
4. The original issue that triggered the skip of macOS has since been
[resolved](https://github.com/actions/runner-images/issues/13341#issuecomment-3681477655).
I also manually ran the affected workflows and confirmed they complete
successfully on macOS. ([`build_maven_java21_macos26.yml`][maven],
[`build_python_3.12_macos26.yml`][python])
[maven]:
https://github.com/nchammas/spark/actions/runs/33208013025/job/98973954067#step:4:468
[python]:
https://github.com/nchammas/spark/actions/runs/33207998706/job/98973904774#step:4:418
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
PR CI:
- CI run here: https://github.com/nchammas/spark/actions/runs/33046412619
- CI run on a baseline branch for comparison:
https://github.com/nchammas/spark/actions/runs/33098805898
As far as the runtimes go, I don't see any notable differences for any
individual job. (Disregard the overall graph runtime since it includes waiting
for runners to pick up the jobs.)
For the scheduled workflows that don't run as part of PR CI tests, I
manually edited them to run on my fork:
- https://github.com/nchammas/spark/actions/runs/33208013025
- https://github.com/nchammas/spark/actions/runs/33207998706
They both ran successfully. (The failure on the extended SQL tests in first
run is unrelated to this PR.)
### Was this patch authored or co-authored using generative AI tooling?
I wrote this with assistance from GitHub Copilot.
Closes #58334 from nchammas/setup-build-env.
Authored-by: Nicholas Chammas <[email protected]>
Signed-off-by: Nicholas Chammas <[email protected]>
---
.github/actions/setup-build-env/action.yml | 134 +++++++++++
.github/workflows/benchmark.yml | 43 +---
.github/workflows/build_and_test.yml | 286 +++---------------------
.github/workflows/build_python_connect.yml | 24 +-
.github/workflows/build_python_connect40.yml | 24 +-
.github/workflows/maven_test.yml | 59 +----
.github/workflows/python_hosted_runner_test.yml | 56 +----
7 files changed, 187 insertions(+), 439 deletions(-)
diff --git a/.github/actions/setup-build-env/action.yml
b/.github/actions/setup-build-env/action.yml
new file mode 100644
index 000000000000..984d738bab54
--- /dev/null
+++ b/.github/actions/setup-build-env/action.yml
@@ -0,0 +1,134 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+name: 'Setup Build Environment'
+description: >-
+ Cache SBT/Maven launcher artifacts, optionally cache or restore Coursier and
+ the Maven local repository, optionally free disk space, and install a Zulu
+ JDK. Callers that use a local composite action must add a bare
+ 'actions/checkout' step first so the latest action definition is present
+ (same bootstrap pattern as checkout-and-sync).
+ There is a default cache limit of 10 GB.
+inputs:
+ java-version:
+ description: >-
+ JDK version passed to actions/setup-java (Zulu distribution). Empty skips
+ the install step (rare; used when the caller installs Java
conditionally).
+ required: false
+ default: ''
+ coursier:
+ description: >-
+ How to handle the Coursier cache: 'cache' (save+restore), 'restore'
+ (restore only), or 'none'.
+ required: false
+ default: 'restore'
+ cache-maven-local:
+ description: 'Cache ~/.m2/repository'
+ required: false
+ default: 'false'
+ free-disk-space:
+ description: >-
+ Free disk space before installing Java. 'host' runs
+ dev/free_disk_space, 'container' runs dev/free_disk_space_container,
+ 'none' skips.
+ required: false
+ default: 'none'
+
+runs:
+ using: 'composite'
+ steps:
+ # Key families are shared across workflows:
+ # SBT: build-<os>-<hash(build files)>
+ # Coursier: coursier-<os>-<hash(**/pom.xml, **/plugins.sbt)>
+ # Maven: maven-<os>-java<ver>-<hash(**/pom.xml)>
+ - name: Cache SBT and Maven
+ uses: actions/cache@v6
+ with:
+ path: |
+ build/apache-maven-*
+ build/*.jar
+ ~/.sbt
+ key: >-
+ build-${{ runner.os }}-${{ hashFiles(
+ '**/pom.xml',
+ 'project/build.properties',
+ 'build/mvn',
+ 'build/sbt',
+ 'build/sbt-launch-lib.bash',
+ 'build/spark-build-info'
+ ) }}
+ restore-keys: |
+ build-${{ runner.os }}-
+
+ - name: Cache Coursier local repository
+ if: inputs.coursier == 'cache'
+ uses: actions/cache@v6
+ with:
+ path: ~/.cache/coursier
+ key: >-
+ coursier-${{ runner.os }}-${{ hashFiles(
+ '**/pom.xml',
+ '**/plugins.sbt'
+ ) }}
+ restore-keys: |
+ coursier-${{ runner.os }}-
+
+ - name: Restore Coursier local repository
+ if: inputs.coursier == 'restore'
+ uses: actions/cache/restore@v6
+ with:
+ path: ~/.cache/coursier
+ key: >-
+ coursier-${{ runner.os }}-${{ hashFiles(
+ '**/pom.xml',
+ '**/plugins.sbt'
+ ) }}
+ restore-keys: |
+ coursier-${{ runner.os }}-
+
+ - name: Cache Maven local repository
+ if: inputs.cache-maven-local == 'true'
+ uses: actions/cache@v6
+ with:
+ path: ~/.m2/repository
+ key: >-
+ maven-${{ runner.os }}-java${{ inputs.java-version }}-${{ hashFiles(
+ '**/pom.xml'
+ ) }}
+ restore-keys: |
+ maven-${{ runner.os }}-java${{ inputs.java-version }}-
+
+ - name: Free up disk space
+ if: inputs.free-disk-space == 'host'
+ continue-on-error: true
+ shell: bash
+ run: timeout --kill-after=30s 10m ./dev/free_disk_space
+
+ - name: Free up disk space
+ if: inputs.free-disk-space == 'container'
+ continue-on-error: true
+ shell: bash
+ run: timeout --kill-after=30s 10m ./dev/free_disk_space_container
+
+ - name: Install Java ${{ inputs.java-version }}
+ if: inputs.java-version != ''
+ uses: actions/setup-java@v6
+ with:
+ distribution: zulu
+ java-version: ${{ inputs.java-version }}
diff --git a/.github/workflows/benchmark.yml b/.github/workflows/benchmark.yml
index e5eea4186f8b..57dee800da08 100644
--- a/.github/workflows/benchmark.yml
+++ b/.github/workflows/benchmark.yml
@@ -93,23 +93,10 @@ jobs:
# In order to get diff files
with:
fetch-depth: 0
- - name: Cache SBT and Maven
- uses: actions/cache@v6
+ - name: Setup build environment
+ uses: ./.github/actions/setup-build-env
with:
- path: |
- build/apache-maven-*
- build/*.jar
- ~/.sbt
- key: build-${{ hashFiles('**/pom.xml', 'project/build.properties',
'build/mvn', 'build/sbt', 'build/sbt-launch-lib.bash',
'build/spark-build-info') }}
- restore-keys: |
- build-
- - name: Cache Coursier local repository
- uses: actions/cache@v6
- with:
- path: ~/.cache/coursier
- key: benchmark-coursier-${{ inputs.jdk }}-${{
hashFiles('**/pom.xml', '**/plugins.sbt') }}
- restore-keys: |
- benchmark-coursier-${{ inputs.jdk }}
+ coursier: cache
- name: Cache TPC-DS generated data
id: cache-tpcds-sf-1
uses: actions/cache@v6
@@ -181,28 +168,11 @@ jobs:
exit 1
fi
fi
- - name: Cache SBT and Maven
- uses: actions/cache@v6
+ - name: Setup build environment
+ uses: ./.github/actions/setup-build-env
with:
- path: |
- build/apache-maven-*
- build/*.jar
- ~/.sbt
- key: build-${{ hashFiles('**/pom.xml', 'project/build.properties',
'build/mvn', 'build/sbt', 'build/sbt-launch-lib.bash',
'build/spark-build-info') }}
- restore-keys: |
- build-
- - name: Cache Coursier local repository
- uses: actions/cache@v6
- with:
- path: ~/.cache/coursier
- key: benchmark-coursier-${{ inputs.jdk }}-${{ hashFiles('**/pom.xml',
'**/plugins.sbt') }}
- restore-keys: |
- benchmark-coursier-${{ inputs.jdk }}
- - name: Install Java ${{ inputs.jdk }}
- uses: actions/setup-java@v6
- with:
- distribution: zulu
java-version: ${{ inputs.jdk }}
+ coursier: cache
- name: Cache TPC-DS generated data
if: inputs.generate-tpcds
id: cache-tpcds-sf-1
@@ -254,4 +224,3 @@ jobs:
with:
name: benchmark-results-${{ inputs.jdk }}-${{ inputs.scala }}-${{
matrix.split }}
path: target/benchmark-results-${{ inputs.jdk }}-${{ inputs.scala
}}.tar
-
diff --git a/.github/workflows/build_and_test.yml
b/.github/workflows/build_and_test.yml
index f9f239114a25..eb5e231b06c2 100644
--- a/.github/workflows/build_and_test.yml
+++ b/.github/workflows/build_and_test.yml
@@ -360,37 +360,11 @@ jobs:
uses: ./.github/actions/checkout-and-sync
with:
ref: ${{ needs.precondition.outputs.head_sha }}
- # Cache local repositories. Note that GitHub Actions cache has a 10G limit.
- - name: Cache SBT and Maven
- uses: actions/cache@v6
- with:
- path: |
- build/apache-maven-*
- build/*.jar
- ~/.sbt
- key: build-${{ runner.os }}-${{ hashFiles('**/pom.xml',
'project/build.properties', 'build/mvn', 'build/sbt',
'build/sbt-launch-lib.bash', 'build/spark-build-info') }}
- restore-keys: |
- build-${{ runner.os }}-
- - name: Restore Coursier local repository
- uses: actions/cache/restore@v6
+ - name: Setup build environment
+ uses: ./.github/actions/setup-build-env
with:
- path: ~/.cache/coursier
- key: coursier-${{ runner.os }}-${{ matrix.java }}-${{ matrix.hadoop
}}-${{ hashFiles('**/pom.xml', '**/plugins.sbt') }}
- restore-keys: |
- coursier-${{ runner.os }}-${{ matrix.java }}-${{ matrix.hadoop }}-
- coursier-${{ runner.os }}-
- - name: Free up disk space
- timeout-minutes: 10
- continue-on-error: true
- run: |
- if [ -f ./dev/free_disk_space ]; then
- ./dev/free_disk_space
- fi
- - name: Install Java ${{ matrix.java }}
- uses: actions/setup-java@v6
- with:
- distribution: zulu
java-version: ${{ matrix.java }}
+ free-disk-space: host
- name: Install Python 3.12
uses: actions/setup-python@v7
# We should install one Python that is higher than 3+ for SQL and Yarn
because:
@@ -563,28 +537,11 @@ jobs:
uses: ./.github/actions/checkout-and-sync
with:
ref: ${{ needs.precondition.outputs.head_sha }}
- - name: Cache SBT and Maven
- uses: actions/cache@v6
- with:
- path: |
- build/apache-maven-*
- build/*.jar
- ~/.sbt
- key: build-${{ runner.os }}-${{ hashFiles('**/pom.xml',
'project/build.properties', 'build/mvn', 'build/sbt',
'build/sbt-launch-lib.bash', 'build/spark-build-info') }}
- restore-keys: |
- build-${{ runner.os }}-
- - name: Cache Coursier local repository
- uses: actions/cache@v6
+ - name: Setup build environment
+ uses: ./.github/actions/setup-build-env
with:
- path: ~/.cache/coursier
- key: coursier-${{ runner.os }}-${{ hashFiles('**/pom.xml',
'**/plugins.sbt') }}
- restore-keys: |
- coursier-${{ runner.os }}-
- - name: Install Java ${{ inputs.java }}
- uses: actions/setup-java@v6
- with:
- distribution: zulu
java-version: ${{ inputs.java }}
+ coursier: cache
- name: Build Spark
run: |
./build/sbt -Phadoop-3 -Pyarn -Pspark-ganglia-lgpl -Phadoop-cloud
-Phive \
@@ -681,34 +638,11 @@ jobs:
with:
ref: ${{ needs.precondition.outputs.head_sha }}
set-safe-directory: 'true'
- # Cache local repositories. Note that GitHub Actions cache has a 10G limit.
- - name: Cache SBT and Maven
- uses: actions/cache@v6
+ - name: Setup build environment
+ uses: ./.github/actions/setup-build-env
with:
- path: |
- build/apache-maven-*
- build/*.jar
- ~/.sbt
- key: build-${{ runner.os }}-${{ hashFiles('**/pom.xml',
'project/build.properties', 'build/mvn', 'build/sbt',
'build/sbt-launch-lib.bash', 'build/spark-build-info') }}
- restore-keys: |
- build-${{ runner.os }}-
- - name: Restore Coursier local repository
- uses: actions/cache/restore@v6
- with:
- path: ~/.cache/coursier
- key: coursier-${{ runner.os }}-${{ hashFiles('**/pom.xml',
'**/plugins.sbt') }}
- restore-keys: |
- coursier-${{ runner.os }}-
- - name: Free up disk space
- timeout-minutes: 10
- continue-on-error: true
- shell: 'script -q -e -c "bash {0}"'
- run: ./dev/free_disk_space_container
- - name: Install Java ${{ matrix.java }}
- uses: actions/setup-java@v6
- with:
- distribution: zulu
java-version: ${{ matrix.java }}
+ free-disk-space: container
- name: List Python packages (${{ env.PYTHON_TO_TEST }})
if: ${{ env.PYTHON_TO_TEST != '' }}
env: ${{ fromJSON(inputs.envs) }}
@@ -838,33 +772,11 @@ jobs:
with:
ref: ${{ needs.precondition.outputs.head_sha }}
set-safe-directory: 'true'
- # Cache local repositories. Note that GitHub Actions cache has a 10G limit.
- - name: Cache SBT and Maven
- uses: actions/cache@v6
- with:
- path: |
- build/apache-maven-*
- build/*.jar
- ~/.sbt
- key: build-${{ runner.os }}-${{ hashFiles('**/pom.xml',
'project/build.properties', 'build/mvn', 'build/sbt',
'build/sbt-launch-lib.bash', 'build/spark-build-info') }}
- restore-keys: |
- build-${{ runner.os }}-
- - name: Restore Coursier local repository
- uses: actions/cache/restore@v6
- with:
- path: ~/.cache/coursier
- key: coursier-${{ runner.os }}-${{ hashFiles('**/pom.xml',
'**/plugins.sbt') }}
- restore-keys: |
- coursier-${{ runner.os }}-
- - name: Free up disk space
- timeout-minutes: 10
- continue-on-error: true
- run: ./dev/free_disk_space_container
- - name: Install Java ${{ inputs.java }}
- uses: actions/setup-java@v6
+ - name: Setup build environment
+ uses: ./.github/actions/setup-build-env
with:
- distribution: zulu
java-version: ${{ inputs.java }}
+ free-disk-space: container
- name: Download precompiled artifact
uses: actions/download-artifact@v8
with:
@@ -957,40 +869,11 @@ jobs:
with:
ref: ${{ needs.precondition.outputs.head_sha }}
set-safe-directory: 'true'
- # Cache local repositories. Note that GitHub Actions cache has a 10G limit.
- - name: Cache SBT and Maven
- uses: actions/cache@v6
- with:
- path: |
- build/apache-maven-*
- build/*.jar
- ~/.sbt
- key: build-${{ runner.os }}-${{ hashFiles('**/pom.xml',
'project/build.properties', 'build/mvn', 'build/sbt',
'build/sbt-launch-lib.bash', 'build/spark-build-info') }}
- restore-keys: |
- build-${{ runner.os }}-
- - name: Restore Coursier local repository
- uses: actions/cache/restore@v6
- with:
- path: ~/.cache/coursier
- key: coursier-${{ runner.os }}-${{ hashFiles('**/pom.xml',
'**/plugins.sbt') }}
- restore-keys: |
- coursier-${{ runner.os }}-
- - name: Cache Maven local repository
- uses: actions/cache@v6
- with:
- path: ~/.m2/repository
- key: docs-maven-${{ runner.os }}-${{ hashFiles('**/pom.xml') }}
- restore-keys: |
- docs-maven-${{ runner.os }}-
- - name: Free up disk space
- timeout-minutes: 10
- continue-on-error: true
- run: ./dev/free_disk_space_container
- - name: Install Java ${{ inputs.java }}
- uses: actions/setup-java@v6
+ - name: Setup build environment
+ uses: ./.github/actions/setup-build-env
with:
- distribution: zulu
java-version: ${{ inputs.java }}
+ free-disk-space: container
- name: License test
run: ./dev/check-license
- name: Dependencies test
@@ -1093,40 +976,11 @@ jobs:
with:
ref: ${{ needs.precondition.outputs.head_sha }}
set-safe-directory: 'true'
- # Cache local repositories. Note that GitHub Actions cache has a 10G limit.
- - name: Cache SBT and Maven
- uses: actions/cache@v6
- with:
- path: |
- build/apache-maven-*
- build/*.jar
- ~/.sbt
- key: build-${{ runner.os }}-${{ hashFiles('**/pom.xml',
'project/build.properties', 'build/mvn', 'build/sbt',
'build/sbt-launch-lib.bash', 'build/spark-build-info') }}
- restore-keys: |
- build-${{ runner.os }}-
- - name: Restore Coursier local repository
- uses: actions/cache/restore@v6
- with:
- path: ~/.cache/coursier
- key: coursier-${{ runner.os }}-${{ hashFiles('**/pom.xml',
'**/plugins.sbt') }}
- restore-keys: |
- coursier-${{ runner.os }}-
- - name: Cache Maven local repository
- uses: actions/cache@v6
- with:
- path: ~/.m2/repository
- key: docs-maven-${{ runner.os }}-${{ hashFiles('**/pom.xml') }}
- restore-keys: |
- docs-maven-${{ runner.os }}-
- - name: Free up disk space
- timeout-minutes: 10
- continue-on-error: true
- run: ./dev/free_disk_space_container
- - name: Install Java ${{ inputs.java }}
- uses: actions/setup-java@v6
+ - name: Setup build environment
+ uses: ./.github/actions/setup-build-env
with:
- distribution: zulu
java-version: ${{ inputs.java }}
+ free-disk-space: container
- name: List Python packages
run: |
lsb_release -a
@@ -1181,27 +1035,9 @@ jobs:
uses: ./.github/actions/checkout-and-sync
with:
ref: ${{ needs.precondition.outputs.head_sha }}
- - name: Cache SBT and Maven
- uses: actions/cache@v6
- with:
- path: |
- build/apache-maven-*
- build/*.jar
- ~/.sbt
- key: build-${{ runner.os }}-${{ hashFiles('**/pom.xml',
'project/build.properties', 'build/mvn', 'build/sbt',
'build/sbt-launch-lib.bash', 'build/spark-build-info') }}
- restore-keys: |
- build-${{ runner.os }}-
- - name: Restore Coursier local repository
- uses: actions/cache/restore@v6
- with:
- path: ~/.cache/coursier
- key: coursier-${{ runner.os }}-${{ hashFiles('**/pom.xml',
'**/plugins.sbt') }}
- restore-keys: |
- coursier-${{ runner.os }}-
- - name: Install Java ${{ inputs.java }}
- uses: actions/setup-java@v6
+ - name: Setup build environment
+ uses: ./.github/actions/setup-build-env
with:
- distribution: zulu
java-version: ${{ inputs.java }}
- name: Download precompiled artifact
uses: actions/download-artifact@v8
@@ -1304,27 +1140,9 @@ jobs:
uses: ./.github/actions/checkout-and-sync
with:
ref: ${{ needs.precondition.outputs.head_sha }}
- - name: Cache SBT and Maven
- uses: actions/cache@v6
- with:
- path: |
- build/apache-maven-*
- build/*.jar
- ~/.sbt
- key: build-${{ runner.os }}-${{ hashFiles('**/pom.xml',
'project/build.properties', 'build/mvn', 'build/sbt',
'build/sbt-launch-lib.bash', 'build/spark-build-info') }}
- restore-keys: |
- build-${{ runner.os }}-
- - name: Restore Coursier local repository
- uses: actions/cache/restore@v6
+ - name: Setup build environment
+ uses: ./.github/actions/setup-build-env
with:
- path: ~/.cache/coursier
- key: coursier-${{ runner.os }}-${{ hashFiles('**/pom.xml',
'**/plugins.sbt') }}
- restore-keys: |
- coursier-${{ runner.os }}-
- - name: Install Java ${{ inputs.java }}
- uses: actions/setup-java@v6
- with:
- distribution: zulu
java-version: ${{ inputs.java }}
- name: Download precompiled artifact
uses: actions/download-artifact@v8
@@ -1375,35 +1193,11 @@ jobs:
uses: ./.github/actions/checkout-and-sync
with:
ref: ${{ needs.precondition.outputs.head_sha }}
- - name: Cache SBT and Maven
- uses: actions/cache@v6
- with:
- path: |
- build/apache-maven-*
- build/*.jar
- ~/.sbt
- key: build-${{ runner.os }}-${{ hashFiles('**/pom.xml',
'project/build.properties', 'build/mvn', 'build/sbt',
'build/sbt-launch-lib.bash', 'build/spark-build-info') }}
- restore-keys: |
- build-${{ runner.os }}-
- - name: Restore Coursier local repository
- uses: actions/cache/restore@v6
+ - name: Setup build environment
+ uses: ./.github/actions/setup-build-env
with:
- path: ~/.cache/coursier
- key: coursier-${{ runner.os }}-${{ hashFiles('**/pom.xml',
'**/plugins.sbt') }}
- restore-keys: |
- coursier-${{ runner.os }}-
- - name: Free up disk space
- timeout-minutes: 10
- continue-on-error: true
- run: |
- if [ -f ./dev/free_disk_space ]; then
- ./dev/free_disk_space
- fi
- - name: Install Java ${{ inputs.java }}
- uses: actions/setup-java@v6
- with:
- distribution: zulu
java-version: ${{ inputs.java }}
+ free-disk-space: host
- name: Download precompiled artifact
uses: actions/download-artifact@v8
with:
@@ -1463,35 +1257,11 @@ jobs:
uses: ./.github/actions/checkout-and-sync
with:
ref: ${{ needs.precondition.outputs.head_sha }}
- - name: Cache SBT and Maven
- uses: actions/cache@v6
- with:
- path: |
- build/apache-maven-*
- build/*.jar
- ~/.sbt
- key: build-${{ runner.os }}-${{ hashFiles('**/pom.xml',
'project/build.properties', 'build/mvn', 'build/sbt',
'build/sbt-launch-lib.bash', 'build/spark-build-info') }}
- restore-keys: |
- build-${{ runner.os }}-
- - name: Restore Coursier local repository
- uses: actions/cache/restore@v6
- with:
- path: ~/.cache/coursier
- key: coursier-${{ runner.os }}-${{ hashFiles('**/pom.xml',
'**/plugins.sbt') }}
- restore-keys: |
- coursier-${{ runner.os }}-
- - name: Free up disk space
- timeout-minutes: 10
- continue-on-error: true
- run: |
- if [ -f ./dev/free_disk_space ]; then
- ./dev/free_disk_space
- fi
- - name: Install Java ${{ inputs.java }}
- uses: actions/setup-java@v6
+ - name: Setup build environment
+ uses: ./.github/actions/setup-build-env
with:
- distribution: zulu
java-version: ${{ inputs.java }}
+ free-disk-space: host
- name: Download precompiled artifact
uses: actions/download-artifact@v8
with:
diff --git a/.github/workflows/build_python_connect.yml
b/.github/workflows/build_python_connect.yml
index 2ae7ce9993e1..b3397c3ffd16 100644
--- a/.github/workflows/build_python_connect.yml
+++ b/.github/workflows/build_python_connect.yml
@@ -34,28 +34,10 @@ jobs:
steps:
- name: Checkout Spark repository
uses: actions/checkout@v7
- - name: Cache SBT and Maven
- uses: actions/cache@v6
+ - name: Setup build environment
+ uses: ./.github/actions/setup-build-env
with:
- path: |
- build/apache-maven-*
- build/*.jar
- ~/.sbt
- key: build-spark-connect-python-only-${{ runner.os }}-${{
hashFiles('**/pom.xml', 'project/build.properties', 'build/mvn', 'build/sbt',
'build/sbt-launch-lib.bash', 'build/spark-build-info') }}
- restore-keys: |
- build-spark-connect-python-only-${{ runner.os }}-
- - name: Cache Coursier local repository
- uses: actions/cache@v6
- with:
- path: ~/.cache/coursier
- key: coursier-build-spark-connect-python-only-${{ runner.os }}-${{
hashFiles('**/pom.xml') }}
- restore-keys: |
- coursier-build-spark-connect-python-only-${{ runner.os }}-
- - name: Install Java 17
- uses: actions/setup-java@v6
- with:
- distribution: zulu
- java-version: 17
+ java-version: '17'
- name: Install Python 3.11
uses: actions/setup-python@v7
with:
diff --git a/.github/workflows/build_python_connect40.yml
b/.github/workflows/build_python_connect40.yml
index fdee2912f9f0..5e259a3a62b6 100644
--- a/.github/workflows/build_python_connect40.yml
+++ b/.github/workflows/build_python_connect40.yml
@@ -36,28 +36,10 @@ jobs:
uses: actions/checkout@v7
with:
fetch-depth: 0
- - name: Cache SBT and Maven
- uses: actions/cache@v6
+ - name: Setup build environment
+ uses: ./.github/actions/setup-build-env
with:
- path: |
- build/apache-maven-*
- build/*.jar
- ~/.sbt
- key: build-spark-connect-python-only-${{ runner.os }}-${{
hashFiles('**/pom.xml', 'project/build.properties', 'build/mvn', 'build/sbt',
'build/sbt-launch-lib.bash', 'build/spark-build-info') }}
- restore-keys: |
- build-spark-connect-python-only-${{ runner.os }}-
- - name: Cache Coursier local repository
- uses: actions/cache@v6
- with:
- path: ~/.cache/coursier
- key: coursier-build-spark-connect-python-only-${{ runner.os }}-${{
hashFiles('**/pom.xml') }}
- restore-keys: |
- coursier-build-spark-connect-python-only-${{ runner.os }}-
- - name: Install Java 17
- uses: actions/setup-java@v6
- with:
- distribution: zulu
- java-version: 17
+ java-version: '17'
- name: Install Python 3.11
uses: actions/setup-python@v7
with:
diff --git a/.github/workflows/maven_test.yml b/.github/workflows/maven_test.yml
index 09fee5e19f4a..47ffbcb7951e 100644
--- a/.github/workflows/maven_test.yml
+++ b/.github/workflows/maven_test.yml
@@ -77,32 +77,12 @@ jobs:
uses: ./.github/actions/checkout-and-sync
with:
ref: ${{ inputs.branch }}
- - name: Cache SBT and Maven
- # TODO(SPARK-54466):
https://github.com/actions/runner-images/issues/13341
- if: ${{ runner.os != 'macOS' }}
- uses: actions/cache@v6
+ - name: Setup build environment
+ uses: ./.github/actions/setup-build-env
with:
- path: |
- build/apache-maven-*
- build/*.jar
- ~/.sbt
- key: build-${{ runner.os }}-${{ hashFiles('**/pom.xml',
'project/build.properties', 'build/mvn', 'build/sbt',
'build/sbt-launch-lib.bash', 'build/spark-build-info') }}
- restore-keys: |
- build-${{ runner.os }}-
- - name: Cache Maven local repository
- # TODO(SPARK-54466):
https://github.com/actions/runner-images/issues/13341
- if: ${{ runner.os != 'macOS' }}
- uses: actions/cache@v6
- with:
- path: ~/.m2/repository
- key: maven-${{ runner.os }}-java${{ inputs.java }}-${{
hashFiles('**/pom.xml') }}
- restore-keys: |
- maven-${{ runner.os }}-java${{ inputs.java }}-
- - name: Install Java ${{ inputs.java }}
- uses: actions/setup-java@v6
- with:
- distribution: zulu
java-version: ${{ inputs.java }}
+ coursier: none
+ cache-maven-local: 'true'
- name: Build Spark with Maven
shell: |
bash -c "if script -qec true 2>/dev/null; then script -qec bash\
{0}; else script -qe /dev/null bash {0}; fi"
@@ -140,8 +120,6 @@ jobs:
needs: precompile-maven
if: (!cancelled())
runs-on: ${{ inputs.os }}
- # TODO(SPARK-54466): https://github.com/actions/runner-images/issues/13341
- # timeout-minutes: 150
strategy:
fail-fast: false
max-parallel: 20
@@ -220,33 +198,12 @@ jobs:
with:
# Fall back to the branch when the precompile job failed before
resolving the SHA.
ref: ${{ needs.precompile-maven.outputs.head_sha || inputs.branch }}
- # Cache local repositories. Note that GitHub Actions cache has a 10G
limit.
- - name: Cache SBT and Maven
- # TODO(SPARK-54466):
https://github.com/actions/runner-images/issues/13341
- if: ${{ runner.os != 'macOS' }}
- uses: actions/cache@v6
- with:
- path: |
- build/apache-maven-*
- build/*.jar
- ~/.sbt
- key: build-${{ runner.os }}-${{ hashFiles('**/pom.xml',
'project/build.properties', 'build/mvn', 'build/sbt',
'build/sbt-launch-lib.bash', 'build/spark-build-info') }}
- restore-keys: |
- build-${{ runner.os }}-
- - name: Cache Maven local repository
- # TODO(SPARK-54466):
https://github.com/actions/runner-images/issues/13341
- if: ${{ runner.os != 'macOS' }}
- uses: actions/cache@v6
- with:
- path: ~/.m2/repository
- key: maven-${{ runner.os }}-java${{ matrix.java }}-${{
hashFiles('**/pom.xml') }}
- restore-keys: |
- maven-${{ runner.os }}-java${{ matrix.java }}-
- - name: Install Java ${{ matrix.java }}
- uses: actions/setup-java@v6
+ - name: Setup build environment
+ uses: ./.github/actions/setup-build-env
with:
- distribution: zulu
java-version: ${{ matrix.java }}
+ coursier: none
+ cache-maven-local: 'true'
- name: Install Python 3.12
uses: actions/setup-python@v7
# We should install one Python that is higher than 3+ for SQL and Yarn
because:
diff --git a/.github/workflows/python_hosted_runner_test.yml
b/.github/workflows/python_hosted_runner_test.yml
index 5150dafaa24b..9ff9ba7f4e09 100644
--- a/.github/workflows/python_hosted_runner_test.yml
+++ b/.github/workflows/python_hosted_runner_test.yml
@@ -81,32 +81,11 @@ jobs:
uses: ./.github/actions/checkout-and-sync
with:
ref: ${{ inputs.branch }}
- - name: Cache SBT and Maven
- # TODO(SPARK-54466):
https://github.com/actions/runner-images/issues/13341
- if: ${{ runner.os != 'macOS' }}
- uses: actions/cache@v6
+ - name: Setup build environment
+ uses: ./.github/actions/setup-build-env
with:
- path: |
- build/apache-maven-*
- build/*.jar
- ~/.sbt
- key: build-${{ runner.os }}-${{ hashFiles('**/pom.xml',
'project/build.properties', 'build/mvn', 'build/sbt',
'build/sbt-launch-lib.bash', 'build/spark-build-info') }}
- restore-keys: |
- build-${{ runner.os }}-
- - name: Cache Coursier local repository
- # TODO(SPARK-54466):
https://github.com/actions/runner-images/issues/13341
- if: ${{ runner.os != 'macOS' }}
- uses: actions/cache@v6
- with:
- path: ~/.cache/coursier
- key: coursier-${{ runner.os }}-${{ hashFiles('**/pom.xml',
'**/plugins.sbt') }}
- restore-keys: |
- coursier-${{ runner.os }}-
- - name: Install Java ${{ inputs.java }}
- uses: actions/setup-java@v6
- with:
- distribution: zulu
java-version: ${{ inputs.java }}
+ coursier: cache
- name: Build Spark
run: |
./build/sbt -Phadoop-3 -Pyarn -Pspark-ganglia-lgpl -Phadoop-cloud
-Phive \
@@ -131,8 +110,6 @@ jobs:
needs: precompile
if: (!cancelled())
runs-on: ${{ inputs.os }}
- # TODO(SPARK-54466): https://github.com/actions/runner-images/issues/13341
- # timeout-minutes: 150
strategy:
fail-fast: false
max-parallel: 20
@@ -185,32 +162,9 @@ jobs:
with:
# Fall back to the branch when the precompile job failed before
resolving the SHA.
ref: ${{ needs.precompile.outputs.head_sha || inputs.branch }}
- # Cache local repositories. Note that GitHub Actions cache has a 10G
limit.
- - name: Cache SBT and Maven
- # TODO(SPARK-54466):
https://github.com/actions/runner-images/issues/13341
- if: ${{ runner.os != 'macOS' }}
- uses: actions/cache@v6
- with:
- path: |
- build/apache-maven-*
- build/*.jar
- ~/.sbt
- key: build-${{ runner.os }}-${{ hashFiles('**/pom.xml',
'project/build.properties', 'build/mvn', 'build/sbt',
'build/sbt-launch-lib.bash', 'build/spark-build-info') }}
- restore-keys: |
- build-${{ runner.os }}-
- - name: Cache Coursier local repository
- # TODO(SPARK-54466):
https://github.com/actions/runner-images/issues/13341
- if: ${{ runner.os != 'macOS' }}
- uses: actions/cache@v6
- with:
- path: ~/.cache/coursier
- key: coursier-${{ runner.os }}-${{ hashFiles('**/pom.xml',
'**/plugins.sbt') }}
- restore-keys: |
- coursier-${{ runner.os }}-
- - name: Install Java ${{ matrix.java }}
- uses: actions/setup-java@v6
+ - name: Setup build environment
+ uses: ./.github/actions/setup-build-env
with:
- distribution: zulu
java-version: ${{ matrix.java }}
- name: Install Python ${{matrix.python}}
uses: actions/setup-python@v7
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
