This is an automated email from the ASF dual-hosted git repository.
nchammas pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new 18c827d84e72 [SPARK-58088][INFRA] Extract checkout-and-sync pattern
into composite GitHub Action
18c827d84e72 is described below
commit 18c827d84e725368a0a77ac6b180164dd2ff8e18
Author: Nicholas Chammas <[email protected]>
AuthorDate: Thu Aug 20 09:26:25 2026 -0400
[SPARK-58088][INFRA] Extract checkout-and-sync pattern into composite
GitHub Action
### What changes were proposed in this pull request?
Extract the widely used checkout-and-sync pattern from various workflows
into a reusable composite action. The pattern is used in about 16 different
places across 3 workflow files.
We first do a normal checkout of the fork so that the latest version of the
composite action is available. Using the composite action, we then checkout
`apache/spark` and merge the changes as we currently do.
With a common action now being used, minor inconsistencies in the
checkout-and-sync pattern have been resolved. Notably, `APACHE_SPARK_REF` is
now always exported to the shell environment. This means that any jobs that
call `dev/run-tests` will always check what files were changed and skip tests
for unchanged modules. This includes `docker-integration-tests`.
There are other opportunities for extracting repeated build patterns. I
will pursue them after proving out this refactoring, which perhaps offers the
highest ROI since this pattern is the most highly used.
### Why are the changes needed?
Having a reusable action will eliminate the need to manually keep so many
chunks of build instructions in sync.
The "checkout fork, then checkout `apache/spark`" approach introduced here
is different from the past attempts made to factor out repeated build steps in
#36924 and #56204.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
The CI build associated with this PR will serve as the test.
### Was this patch authored or co-authored using generative AI tooling?
I wrote this with assistance from GitHub Copilot.
Closes #57187 from nchammas/SPARK-58088-checkout-sync-composite-action.
Authored-by: Nicholas Chammas <[email protected]>
Signed-off-by: Nicholas Chammas <[email protected]>
(cherry picked from commit 44093317fb16ff5fe22dbf0943f5eeea2c31a7be)
Signed-off-by: Nicholas Chammas <[email protected]>
---
.github/actions/checkout-and-sync/action.yml | 68 +++++++++
.github/workflows/build_and_test.yml | 180 ++++++------------------
.github/workflows/maven_test.yml | 33 ++---
.github/workflows/python_hosted_runner_test.yml | 33 ++---
4 files changed, 126 insertions(+), 188 deletions(-)
diff --git a/.github/actions/checkout-and-sync/action.yml
b/.github/actions/checkout-and-sync/action.yml
new file mode 100644
index 000000000000..8481678b5115
--- /dev/null
+++ b/.github/actions/checkout-and-sync/action.yml
@@ -0,0 +1,68 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+name: 'Checkout and Sync'
+description: >-
+ Check out apache/spark at a pinned ref and, on forks, squash-merge the fork
+ branch on top so the build tests the combined result. Callers must add a
+ bare 'actions/checkout' step before invoking this action so that the latest
+ action definition is present in the workspace.
+inputs:
+ ref:
+ description: 'Git ref to check out from apache/spark'
+ required: true
+ set-safe-directory:
+ description: 'Add GITHUB_WORKSPACE to git safe.directory (needed inside
containers)'
+ required: false
+ default: 'false'
+outputs:
+ head_sha:
+ description: 'The apache/spark HEAD SHA checked out (before the fork
merge)'
+ value: ${{ steps.resolve-sha.outputs.head_sha }}
+runs:
+ using: 'composite'
+ steps:
+ - name: Checkout Spark repository
+ uses: actions/checkout@v6
+ with:
+ fetch-depth: 0
+ repository: apache/spark
+ ref: ${{ inputs.ref }}
+ - name: Add GITHUB_WORKSPACE to git trust safe.directory
+ if: inputs.set-safe-directory == 'true'
+ shell: bash
+ run: git config --global --add safe.directory ${GITHUB_WORKSPACE}
+ - name: Resolve apache/spark HEAD SHA
+ id: resolve-sha
+ shell: bash
+ run: echo "head_sha=$(git rev-parse HEAD)" >> $GITHUB_OUTPUT
+ - name: Sync the current branch with the latest in Apache Spark
+ if: github.repository != 'apache/spark'
+ shell: bash
+ run: |
+ echo "APACHE_SPARK_REF=$(git rev-parse HEAD)" >> $GITHUB_ENV
+ git fetch https://github.com/$GITHUB_REPOSITORY.git
${GITHUB_REF#refs/heads/}
+ git \
+ -c user.name='Apache Spark Test Account' \
+ -c user.email='[email protected]' \
+ merge --no-commit --progress --squash FETCH_HEAD
+ git \
+ -c user.name='Apache Spark Test Account' \
+ -c user.email='[email protected]' \
+ commit -m "Merged commit" --allow-empty
diff --git a/.github/workflows/build_and_test.yml
b/.github/workflows/build_and_test.yml
index 64a722ddbb36..365e2bbbdbbf 100644
--- a/.github/workflows/build_and_test.yml
+++ b/.github/workflows/build_and_test.yml
@@ -66,7 +66,7 @@ jobs:
outputs:
required: ${{ steps.set-outputs.outputs.required }}
# Pinned so every downstream job checks out the same snapshot, even if
`master` advances mid-run.
- head_sha: ${{ steps.resolve-sha.outputs.head_sha }}
+ head_sha: ${{ steps.checkout.outputs.head_sha }}
image_docs_url: ${{
steps.infra-image-docs-outputs.outputs.image_docs_url }}
image_docs_url_link: ${{
steps.infra-image-link.outputs.image_docs_url_link }}
image_lint_url: ${{
steps.infra-image-lint-outputs.outputs.image_lint_url }}
@@ -76,22 +76,13 @@ jobs:
image_pyspark_url: ${{
steps.infra-image-pyspark-outputs.outputs.image_pyspark_url }}
image_pyspark_url_link: ${{
steps.infra-image-link.outputs.image_pyspark_url_link }}
steps:
- - name: Checkout Spark repository
+ - name: Bootstrap composite actions
uses: actions/checkout@v6
+ - name: Checkout and sync Spark repository
+ id: checkout
+ uses: ./.github/actions/checkout-and-sync
with:
- fetch-depth: 0
- repository: apache/spark
ref: ${{ inputs.branch }}
- - name: Resolve apache/spark HEAD SHA
- id: resolve-sha
- run: echo "head_sha=$(git rev-parse HEAD)" >> $GITHUB_OUTPUT
- - name: Sync the current branch with the latest in Apache Spark
- if: github.repository != 'apache/spark'
- run: |
- echo "APACHE_SPARK_REF=$(git rev-parse HEAD)" >> $GITHUB_ENV
- git fetch https://github.com/$GITHUB_REPOSITORY.git
${GITHUB_REF#refs/heads/}
- git -c user.name='Apache Spark Test Account' -c
user.email='[email protected]' merge --no-commit --progress --squash
FETCH_HEAD
- git -c user.name='Apache Spark Test Account' -c
user.email='[email protected]' commit -m "Merged commit" --allow-empty
- name: Check all modules
id: set-outputs
run: |
@@ -340,20 +331,12 @@ jobs:
SKIP_MIMA: true
SKIP_PACKAGING: true
steps:
- - name: Checkout Spark repository
+ - name: Bootstrap composite actions
uses: actions/checkout@v6
- # In order to fetch changed files
+ - name: Checkout and sync Spark repository
+ uses: ./.github/actions/checkout-and-sync
with:
- fetch-depth: 0
- repository: apache/spark
ref: ${{ needs.precondition.outputs.head_sha }}
- - name: Sync the current branch with the latest in Apache Spark
- if: github.repository != 'apache/spark'
- run: |
- echo "APACHE_SPARK_REF=$(git rev-parse HEAD)" >> $GITHUB_ENV
- git fetch https://github.com/$GITHUB_REPOSITORY.git
${GITHUB_REF#refs/heads/}
- git -c user.name='Apache Spark Test Account' -c
user.email='[email protected]' merge --no-commit --progress --squash
FETCH_HEAD
- git -c user.name='Apache Spark Test Account' -c
user.email='[email protected]' commit -m "Merged commit" --allow-empty
# Cache local repositories. Note that GitHub Actions cache has a 10G limit.
- name: Cache SBT and Maven
uses: actions/cache@v5
@@ -467,20 +450,12 @@ jobs:
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- - name: Checkout Spark repository
+ - name: Bootstrap composite actions
uses: actions/checkout@v6
- # In order to fetch changed files
+ - name: Checkout and sync Spark repository
+ uses: ./.github/actions/checkout-and-sync
with:
- fetch-depth: 0
- repository: apache/spark
ref: ${{ needs.precondition.outputs.head_sha }}
- - name: Sync the current branch with the latest in Apache Spark
- if: github.repository != 'apache/spark'
- run: |
- echo "APACHE_SPARK_REF=$(git rev-parse HEAD)" >> $GITHUB_ENV
- git fetch https://github.com/$GITHUB_REPOSITORY.git
${GITHUB_REF#refs/heads/}
- git -c user.name='Apache Spark Test Account' -c
user.email='[email protected]' merge --no-commit --progress --squash
FETCH_HEAD
- git -c user.name='Apache Spark Test Account' -c
user.email='[email protected]' commit -m "Merged commit" --allow-empty
- name: Set up QEMU
uses: docker/setup-qemu-action@96fe6ef7f33517b61c61be40b68a1882f3264fb8
- name: Set up Docker Buildx
@@ -558,19 +533,12 @@ jobs:
HIVE_PROFILE: hive2.3
GITHUB_PREV_SHA: ${{ github.event.before }}
steps:
- - name: Checkout Spark repository
+ - name: Bootstrap composite actions
uses: actions/checkout@v6
+ - name: Checkout and sync Spark repository
+ uses: ./.github/actions/checkout-and-sync
with:
- fetch-depth: 0
- repository: apache/spark
ref: ${{ needs.precondition.outputs.head_sha }}
- - name: Sync the current branch with the latest in Apache Spark
- if: github.repository != 'apache/spark'
- run: |
- echo "APACHE_SPARK_REF=$(git rev-parse HEAD)" >> $GITHUB_ENV
- git fetch https://github.com/$GITHUB_REPOSITORY.git
${GITHUB_REF#refs/heads/}
- git -c user.name='Apache Spark Test Account' -c
user.email='[email protected]' merge --no-commit --progress --squash
FETCH_HEAD
- git -c user.name='Apache Spark Test Account' -c
user.email='[email protected]' commit -m "Merged commit" --allow-empty
- name: Cache SBT and Maven
uses: actions/cache@v5
with:
@@ -682,23 +650,13 @@ jobs:
# Cap CI fuzz volume; the explicit @example edge seeds still always run.
RUN_HYPOTHESIS_MAX_EXAMPLES: 50
steps:
- - name: Checkout Spark repository
+ - name: Bootstrap composite actions
uses: actions/checkout@v6
- # In order to fetch changed files
+ - name: Checkout and sync Spark repository
+ uses: ./.github/actions/checkout-and-sync
with:
- fetch-depth: 0
- repository: apache/spark
ref: ${{ needs.precondition.outputs.head_sha }}
- - name: Add GITHUB_WORKSPACE to git trust safe.directory
- run: |
- git config --global --add safe.directory ${GITHUB_WORKSPACE}
- - name: Sync the current branch with the latest in Apache Spark
- if: github.repository != 'apache/spark'
- run: |
- echo "APACHE_SPARK_REF=$(git rev-parse HEAD)" >> $GITHUB_ENV
- git fetch https://github.com/$GITHUB_REPOSITORY.git
${GITHUB_REF#refs/heads/}
- git -c user.name='Apache Spark Test Account' -c
user.email='[email protected]' merge --no-commit --progress --squash
FETCH_HEAD
- git -c user.name='Apache Spark Test Account' -c
user.email='[email protected]' commit -m "Merged commit" --allow-empty
+ set-safe-directory: 'true'
# Cache local repositories. Note that GitHub Actions cache has a 10G limit.
- name: Cache SBT and Maven
uses: actions/cache@v5
@@ -849,23 +807,13 @@ jobs:
SKIP_MIMA: true
SKIP_PACKAGING: true
steps:
- - name: Checkout Spark repository
+ - name: Bootstrap composite actions
uses: actions/checkout@v6
- # In order to fetch changed files
+ - name: Checkout and sync Spark repository
+ uses: ./.github/actions/checkout-and-sync
with:
- fetch-depth: 0
- repository: apache/spark
ref: ${{ needs.precondition.outputs.head_sha }}
- - name: Add GITHUB_WORKSPACE to git trust safe.directory
- run: |
- git config --global --add safe.directory ${GITHUB_WORKSPACE}
- - name: Sync the current branch with the latest in Apache Spark
- if: github.repository != 'apache/spark'
- run: |
- echo "APACHE_SPARK_REF=$(git rev-parse HEAD)" >> $GITHUB_ENV
- git fetch https://github.com/$GITHUB_REPOSITORY.git
${GITHUB_REF#refs/heads/}
- git -c user.name='Apache Spark Test Account' -c
user.email='[email protected]' merge --no-commit --progress --squash
FETCH_HEAD
- git -c user.name='Apache Spark Test Account' -c
user.email='[email protected]' commit -m "Merged commit" --allow-empty
+ set-safe-directory: 'true'
# Cache local repositories. Note that GitHub Actions cache has a 10G limit.
- name: Cache SBT and Maven
uses: actions/cache@v5
@@ -933,18 +881,12 @@ jobs:
name: Protobuf breaking change detection and Python CodeGen check
runs-on: ubuntu-latest
steps:
- - name: Checkout Spark repository
+ - name: Bootstrap composite actions
uses: actions/checkout@v6
+ - name: Checkout and sync Spark repository
+ uses: ./.github/actions/checkout-and-sync
with:
- fetch-depth: 0
- repository: apache/spark
ref: ${{ needs.precondition.outputs.head_sha }}
- - name: Sync the current branch with the latest in Apache Spark
- if: github.repository != 'apache/spark'
- run: |
- git fetch https://github.com/$GITHUB_REPOSITORY.git
${GITHUB_REF#refs/heads/}
- git -c user.name='Apache Spark Test Account' -c
user.email='[email protected]' merge --no-commit --progress --squash
FETCH_HEAD
- git -c user.name='Apache Spark Test Account' -c
user.email='[email protected]' commit -m "Merged commit" --allow-empty
- name: Install Buf
uses: bufbuild/buf-setup-action@a47c93e0b1648d5651a065437926377d060baa99
# v1
with:
@@ -1033,22 +975,13 @@ jobs:
container:
image: ${{ needs.precondition.outputs.image_lint_url_link }}
steps:
- - name: Checkout Spark repository
+ - name: Bootstrap composite actions
uses: actions/checkout@v6
+ - name: Checkout and sync Spark repository
+ uses: ./.github/actions/checkout-and-sync
with:
- fetch-depth: 0
- repository: apache/spark
ref: ${{ needs.precondition.outputs.head_sha }}
- - name: Add GITHUB_WORKSPACE to git trust safe.directory
- run: |
- git config --global --add safe.directory ${GITHUB_WORKSPACE}
- - name: Sync the current branch with the latest in Apache Spark
- if: github.repository != 'apache/spark'
- run: |
- echo "APACHE_SPARK_REF=$(git rev-parse HEAD)" >> $GITHUB_ENV
- git fetch https://github.com/$GITHUB_REPOSITORY.git
${GITHUB_REF#refs/heads/}
- git -c user.name='Apache Spark Test Account' -c
user.email='[email protected]' merge --no-commit --progress --squash
FETCH_HEAD
- git -c user.name='Apache Spark Test Account' -c
user.email='[email protected]' commit -m "Merged commit" --allow-empty
+ set-safe-directory: 'true'
# Cache local repositories. Note that GitHub Actions cache has a 10G limit.
- name: Cache SBT and Maven
uses: actions/cache@v5
@@ -1169,22 +1102,13 @@ jobs:
container:
image: ${{ needs.precondition.outputs.image_docs_url_link }}
steps:
- - name: Checkout Spark repository
+ - name: Bootstrap composite actions
uses: actions/checkout@v6
+ - name: Checkout and sync Spark repository
+ uses: ./.github/actions/checkout-and-sync
with:
- fetch-depth: 0
- repository: apache/spark
ref: ${{ needs.precondition.outputs.head_sha }}
- - name: Add GITHUB_WORKSPACE to git trust safe.directory
- run: |
- git config --global --add safe.directory ${GITHUB_WORKSPACE}
- - name: Sync the current branch with the latest in Apache Spark
- if: github.repository != 'apache/spark'
- run: |
- echo "APACHE_SPARK_REF=$(git rev-parse HEAD)" >> $GITHUB_ENV
- git fetch https://github.com/$GITHUB_REPOSITORY.git
${GITHUB_REF#refs/heads/}
- git -c user.name='Apache Spark Test Account' -c
user.email='[email protected]' merge --no-commit --progress --squash
FETCH_HEAD
- git -c user.name='Apache Spark Test Account' -c
user.email='[email protected]' commit -m "Merged commit" --allow-empty
+ set-safe-directory: 'true'
# Cache local repositories. Note that GitHub Actions cache has a 10G limit.
- name: Cache SBT and Maven
uses: actions/cache@v5
@@ -1275,18 +1199,12 @@ jobs:
env:
SPARK_LOCAL_IP: localhost
steps:
- - name: Checkout Spark repository
+ - name: Bootstrap composite actions
uses: actions/checkout@v6
+ - name: Checkout and sync Spark repository
+ uses: ./.github/actions/checkout-and-sync
with:
- fetch-depth: 0
- repository: apache/spark
ref: ${{ needs.precondition.outputs.head_sha }}
- - name: Sync the current branch with the latest in Apache Spark
- if: github.repository != 'apache/spark'
- run: |
- git fetch https://github.com/$GITHUB_REPOSITORY.git
${GITHUB_REF#refs/heads/}
- git -c user.name='Apache Spark Test Account' -c
user.email='[email protected]' merge --no-commit --progress --squash
FETCH_HEAD
- git -c user.name='Apache Spark Test Account' -c
user.email='[email protected]' commit -m "Merged commit" --allow-empty
- name: Cache SBT and Maven
uses: actions/cache@v5
with:
@@ -1399,19 +1317,12 @@ jobs:
SKIP_MIMA: true
SKIP_PACKAGING: true
steps:
- - name: Checkout Spark repository
+ - name: Bootstrap composite actions
uses: actions/checkout@v6
+ - name: Checkout and sync Spark repository
+ uses: ./.github/actions/checkout-and-sync
with:
- fetch-depth: 0
- repository: apache/spark
ref: ${{ needs.precondition.outputs.head_sha }}
- - name: Sync the current branch with the latest in Apache Spark
- if: github.repository != 'apache/spark'
- run: |
- echo "APACHE_SPARK_REF=$(git rev-parse HEAD)" >> $GITHUB_ENV
- git fetch https://github.com/$GITHUB_REPOSITORY.git
${GITHUB_REF#refs/heads/}
- git -c user.name='Apache Spark Test Account' -c
user.email='[email protected]' merge --no-commit --progress --squash
FETCH_HEAD
- git -c user.name='Apache Spark Test Account' -c
user.email='[email protected]' commit -m "Merged commit" --allow-empty
- name: Cache SBT and Maven
uses: actions/cache@v5
with:
@@ -1477,19 +1388,12 @@ jobs:
runs-on: ubuntu-latest
timeout-minutes: 120
steps:
- - name: Checkout Spark repository
+ - name: Bootstrap composite actions
uses: actions/checkout@v6
+ - name: Checkout and sync Spark repository
+ uses: ./.github/actions/checkout-and-sync
with:
- fetch-depth: 0
- repository: apache/spark
ref: ${{ needs.precondition.outputs.head_sha }}
- - name: Sync the current branch with the latest in Apache Spark
- if: github.repository != 'apache/spark'
- run: |
- echo "APACHE_SPARK_REF=$(git rev-parse HEAD)" >> $GITHUB_ENV
- git fetch https://github.com/$GITHUB_REPOSITORY.git
${GITHUB_REF#refs/heads/}
- git -c user.name='Apache Spark Test Account' -c
user.email='[email protected]' merge --no-commit --progress --squash
FETCH_HEAD
- git -c user.name='Apache Spark Test Account' -c
user.email='[email protected]' commit -m "Merged commit" --allow-empty
- name: Cache SBT and Maven
uses: actions/cache@v5
with:
diff --git a/.github/workflows/maven_test.yml b/.github/workflows/maven_test.yml
index fa7cdd20d6ba..563461c6a8f9 100644
--- a/.github/workflows/maven_test.yml
+++ b/.github/workflows/maven_test.yml
@@ -63,29 +63,20 @@ jobs:
continue-on-error: true
outputs:
# Pinned so the build job checks out the same snapshot, even if the
branch advances mid-run.
- head_sha: ${{ steps.resolve-sha.outputs.head_sha }}
+ head_sha: ${{ steps.checkout.outputs.head_sha }}
env:
HADOOP_PROFILE: ${{ inputs.hadoop }}
HIVE_PROFILE: hive2.3
SPARK_LOCAL_IP: localhost
GITHUB_PREV_SHA: ${{ github.event.before }}
steps:
- - name: Checkout Spark repository
+ - name: Bootstrap composite actions
uses: actions/checkout@v6
+ - name: Checkout and sync Spark repository
+ id: checkout
+ uses: ./.github/actions/checkout-and-sync
with:
- fetch-depth: 0
- repository: apache/spark
ref: ${{ inputs.branch }}
- - name: Resolve apache/spark HEAD SHA
- id: resolve-sha
- run: echo "head_sha=$(git rev-parse HEAD)" >> $GITHUB_OUTPUT
- - name: Sync the current branch with the latest in Apache Spark
- if: github.repository != 'apache/spark'
- run: |
- echo "APACHE_SPARK_REF=$(git rev-parse HEAD)" >> $GITHUB_ENV
- git fetch https://github.com/$GITHUB_REPOSITORY.git
${GITHUB_REF#refs/heads/}
- git -c user.name='Apache Spark Test Account' -c
user.email='[email protected]' merge --no-commit --progress --squash
FETCH_HEAD
- git -c user.name='Apache Spark Test Account' -c
user.email='[email protected]' commit -m "Merged commit" --allow-empty
- name: Cache SBT and Maven
# TODO(SPARK-54466):
https://github.com/actions/runner-images/issues/13341
if: ${{ runner.os != 'macOS' }}
@@ -222,21 +213,13 @@ jobs:
SPARK_LOCAL_IP: localhost
GITHUB_PREV_SHA: ${{ github.event.before }}
steps:
- - name: Checkout Spark repository
+ - name: Bootstrap composite actions
uses: actions/checkout@v6
- # In order to fetch changed files
+ - name: Checkout and sync Spark repository
+ uses: ./.github/actions/checkout-and-sync
with:
- fetch-depth: 0
- repository: apache/spark
# Fall back to the branch when the precompile job failed before
resolving the SHA.
ref: ${{ needs.precompile-maven.outputs.head_sha || inputs.branch }}
- - name: Sync the current branch with the latest in Apache Spark
- if: github.repository != 'apache/spark'
- run: |
- echo "APACHE_SPARK_REF=$(git rev-parse HEAD)" >> $GITHUB_ENV
- git fetch https://github.com/$GITHUB_REPOSITORY.git
${GITHUB_REF#refs/heads/}
- git -c user.name='Apache Spark Test Account' -c
user.email='[email protected]' merge --no-commit --progress --squash
FETCH_HEAD
- git -c user.name='Apache Spark Test Account' -c
user.email='[email protected]' commit -m "Merged commit" --allow-empty
# Cache local repositories. Note that GitHub Actions cache has a 10G
limit.
- name: Cache SBT and Maven
# TODO(SPARK-54466):
https://github.com/actions/runner-images/issues/13341
diff --git a/.github/workflows/python_hosted_runner_test.yml
b/.github/workflows/python_hosted_runner_test.yml
index f7e7ded71ce0..440e27f39f5f 100644
--- a/.github/workflows/python_hosted_runner_test.yml
+++ b/.github/workflows/python_hosted_runner_test.yml
@@ -67,29 +67,20 @@ jobs:
continue-on-error: true
outputs:
# Pinned so the build job checks out the same snapshot, even if the
branch advances mid-run.
- head_sha: ${{ steps.resolve-sha.outputs.head_sha }}
+ head_sha: ${{ steps.checkout.outputs.head_sha }}
env:
HADOOP_PROFILE: ${{ inputs.hadoop }}
HIVE_PROFILE: hive2.3
SPARK_LOCAL_IP: localhost
GITHUB_PREV_SHA: ${{ github.event.before }}
steps:
- - name: Checkout Spark repository
+ - name: Bootstrap composite actions
uses: actions/checkout@v6
+ - name: Checkout and sync Spark repository
+ id: checkout
+ uses: ./.github/actions/checkout-and-sync
with:
- fetch-depth: 0
- repository: apache/spark
ref: ${{ inputs.branch }}
- - name: Resolve apache/spark HEAD SHA
- id: resolve-sha
- run: echo "head_sha=$(git rev-parse HEAD)" >> $GITHUB_OUTPUT
- - name: Sync the current branch with the latest in Apache Spark
- if: github.repository != 'apache/spark'
- run: |
- echo "APACHE_SPARK_REF=$(git rev-parse HEAD)" >> $GITHUB_ENV
- git fetch https://github.com/$GITHUB_REPOSITORY.git
${GITHUB_REF#refs/heads/}
- git -c user.name='Apache Spark Test Account' -c
user.email='[email protected]' merge --no-commit --progress --squash
FETCH_HEAD
- git -c user.name='Apache Spark Test Account' -c
user.email='[email protected]' commit -m "Merged commit" --allow-empty
- name: Cache SBT and Maven
# TODO(SPARK-54466):
https://github.com/actions/runner-images/issues/13341
if: ${{ runner.os != 'macOS' }}
@@ -185,21 +176,13 @@ jobs:
BRANCH: ${{ inputs.branch }}
PYSPARK_TEST_TIMEOUT: 450
steps:
- - name: Checkout Spark repository
+ - name: Bootstrap composite actions
uses: actions/checkout@v6
- # In order to fetch changed files
+ - name: Checkout and sync Spark repository
+ uses: ./.github/actions/checkout-and-sync
with:
- fetch-depth: 0
- repository: apache/spark
# Fall back to the branch when the precompile job failed before
resolving the SHA.
ref: ${{ needs.precompile.outputs.head_sha || inputs.branch }}
- - name: Sync the current branch with the latest in Apache Spark
- if: github.repository != 'apache/spark'
- run: |
- echo "APACHE_SPARK_REF=$(git rev-parse HEAD)" >> $GITHUB_ENV
- git fetch https://github.com/$GITHUB_REPOSITORY.git
${GITHUB_REF#refs/heads/}
- git -c user.name='Apache Spark Test Account' -c
user.email='[email protected]' merge --no-commit --progress --squash
FETCH_HEAD
- git -c user.name='Apache Spark Test Account' -c
user.email='[email protected]' commit -m "Merged commit" --allow-empty
# Cache local repositories. Note that GitHub Actions cache has a 10G
limit.
- name: Cache SBT and Maven
# TODO(SPARK-54466):
https://github.com/actions/runner-images/issues/13341
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
