This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 3bf07190510 [SPARK-39735][INFRA] Move image condition to jobs to make
non-master schedule job work
3bf07190510 is described below
commit 3bf07190510e1dc9a882914204995a9b63a79d60
Author: Yikun Jiang <[email protected]>
AuthorDate: Mon Jul 11 22:08:07 2022 +0900
[SPARK-39735][INFRA] Move image condition to jobs to make non-master
schedule job work
### What changes were proposed in this pull request?
Currently, we only enable the infra-image when `input.branches == master`,
otherwise infra-image will skip, and branches schedule job will failed due to
empty container name.
### Why are the changes needed?
We need to move image condition to jobs to make branch schedule job work.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
CI passed
- base image push the [right
image](https://github.com/Yikun/spark/runs/7280303774?check_suite_focus=true#step:7:209
)
- precondition generate the [right image
url](https://github.com/Yikun/spark/runs/7280249393?check_suite_focus=true#step:5:4)
- pyspark use [right image
url](https://github.com/Yikun/spark/runs/7280366394?check_suite_focus=true#step:2:19)
- lint use [right
image:](https://github.com/Yikun/spark/runs/7280366124?check_suite_focus=true#step:2:19)
Closes #37155 from Yikun/SPARK-39736-precondition.
Authored-by: Yikun Jiang <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
---
.github/workflows/build_and_test.yml | 33 ++++++++++++++++-----------------
1 file changed, 16 insertions(+), 17 deletions(-)
diff --git a/.github/workflows/build_and_test.yml
b/.github/workflows/build_and_test.yml
index febc6383c2e..4949c6159d4 100644
--- a/.github/workflows/build_and_test.yml
+++ b/.github/workflows/build_and_test.yml
@@ -56,6 +56,11 @@ jobs:
GITHUB_PREV_SHA: ${{ github.event.before }}
outputs:
required: ${{ steps.set-outputs.outputs.required }}
+ image_url: >-
+ ${{
+ (inputs.branch == 'master' &&
steps.infra-image-outputs.outputs.image_url)
+ || 'dongjoon/apache-spark-github-action-image:20220207'
+ }}
steps:
- name: Checkout Spark repository
uses: actions/checkout@v2
@@ -108,6 +113,14 @@ jobs:
precondition="${precondition//$'\n'/'%0A'}"
echo "::set-output name=required::$precondition"
fi
+ - name: Generate infra image URL
+ id: infra-image-outputs
+ run: |
+ # Convert to lowercase to meet Docker repo name requirement
+ REPO_OWNER=$(echo "${{ github.repository_owner }}" | tr '[:upper:]'
'[:lower:]')
+ IMG_NAME="apache-spark-ci-image:${{ inputs.branch }}-${{ github.run_id
}}"
+ IMG_URL="ghcr.io/$REPO_OWNER/$IMG_NAME"
+ echo ::set-output name=image_url::$IMG_URL
# Build: build Spark and run the tests for specified modules.
build:
@@ -260,21 +273,7 @@ jobs:
fromJson(needs.precondition.outputs.required).lint == 'true') &&
inputs.branch == 'master'
runs-on: ubuntu-latest
- outputs:
- image_url: >-
- ${{
- (inputs.branch == 'master' &&
steps.infra-image-outputs.outputs.image_url)
- || 'dongjoon/apache-spark-github-action-image:20220207'
- }}
steps:
- - name: Generate image name and url
- id: infra-image-outputs
- run: |
- # Convert to lowercase to meet docker repo name requirement
- REPO_OWNER=$(echo "${{ github.repository_owner }}" | tr '[:upper:]'
'[:lower:]')
- IMG_NAME="apache-spark-ci-image:${{ inputs.branch }}-${{
github.run_id }}"
- IMG_URL="ghcr.io/$REPO_OWNER/$IMG_NAME"
- echo ::set-output name=image_url::$IMG_URL
- name: Login to GitHub Container Registry
uses: docker/login-action@v2
with:
@@ -306,7 +305,7 @@ jobs:
context: ./dev/infra/
push: true
tags: |
- ${{ steps.infra-image-outputs.outputs.image_url }}
+ ${{ needs.precondition.outputs.image_url }}
# Use the infra image cache to speed up
cache-from:
type=registry,ref=ghcr.io/apache/spark/apache-spark-github-action-image-cache:${{
inputs.branch }}
@@ -317,7 +316,7 @@ jobs:
name: "Build modules: ${{ matrix.modules }}"
runs-on: ubuntu-20.04
container:
- image: ${{ needs.infra-image.outputs.image_url }}
+ image: ${{ needs.precondition.outputs.image_url }}
strategy:
fail-fast: false
matrix:
@@ -498,7 +497,7 @@ jobs:
PYSPARK_DRIVER_PYTHON: python3.9
PYSPARK_PYTHON: python3.9
container:
- image: ${{ needs.infra-image.outputs.image_url }}
+ image: ${{ needs.precondition.outputs.image_url }}
steps:
- name: Checkout Spark repository
uses: actions/checkout@v2
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
