This is an automated email from the ASF dual-hosted git repository.
zhengruifeng pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new 16a2797a5c81 [SPARK-58285][INFRA] Remove stale container CI dependency
bypasses
16a2797a5c81 is described below
commit 16a2797a5c813b32da2d1f85045ab656f38bdf1a
Author: Ruifeng Zheng <[email protected]>
AuthorDate: Thu Jul 23 20:40:01 2026 +0800
[SPARK-58285][INFRA] Remove stale container CI dependency bypasses
### What changes were proposed in this pull request?
Removes the cancelled-status dependency bypasses from the PySpark, SparkR,
lint, and documentation jobs. Replaces the PySpark matrix's obsolete non-master
fallback comments with descriptions of the entries selected by each change gate.
### Why are the changes needed?
The bypasses were introduced when non-master jobs used static fallback
container images. These jobs now use per-run images produced by infra-image.
When that build fails, the bypass still schedules the entire PySpark matrix and
the SparkR, lint, and documentation jobs; they then fail minutes later while
trying to start from images that were never pushed. No dependent container job
can run successfully in that state, so GitHub Actions should block them until
infra-image succeeds.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Ran git diff --check. No test suite was run because this is a GitHub
Actions job-condition and comment change.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Codex (GPT-5)
Closes #57457 from zhengruifeng/remove-pyspark-needs-bypass-dev4.
Authored-by: Ruifeng Zheng <[email protected]>
Signed-off-by: Ruifeng Zheng <[email protected]>
(cherry picked from commit 3ce5de6baa381f1ee07b3d334a8c5ec3e06dbc25)
Signed-off-by: Ruifeng Zheng <[email protected]>
---
.github/workflows/build_and_test.yml | 23 ++++++-----------------
1 file changed, 6 insertions(+), 17 deletions(-)
diff --git a/.github/workflows/build_and_test.yml
b/.github/workflows/build_and_test.yml
index 78cc56082cc7..b81592650aac 100644
--- a/.github/workflows/build_and_test.yml
+++ b/.github/workflows/build_and_test.yml
@@ -325,9 +325,6 @@ jobs:
excluded-tags:
org.apache.spark.tags.ExtendedSQLTest,org.apache.spark.tags.SlowSQLTest
comment: "- other tests"
exclude:
- # Always run if yarn == 'true', even infra-image is skip (such as
non-master job)
- # In practice, the build will run in individual PR, but not against
the individual commit
- # in Apache Spark repository.
- modules: ${{ fromJson(needs.precondition.outputs.required).yarn !=
'true' && 'yarn' }}
# Skip the core/utils group when a PR doesn't touch those modules
(e.g. SQL-only or
# PySpark-only changes). The precondition is opt-out: omitted means
run (so periodic
@@ -622,8 +619,9 @@ jobs:
pyspark:
needs: [precondition, infra-image, precompile]
- # always run if pyspark == 'true', even infra-image is skip (such as
non-master job)
- if: (!cancelled()) &&
(fromJson(needs.precondition.outputs.required).pyspark == 'true' ||
fromJson(needs.precondition.outputs.required).pyspark-pandas == 'true')
+ if: >-
+ fromJson(needs.precondition.outputs.required).pyspark == 'true' ||
+ fromJson(needs.precondition.outputs.required).pyspark-pandas == 'true'
name: "Build modules: ${{ matrix.modules }}"
runs-on: ubuntu-latest
timeout-minutes: 120
@@ -660,9 +658,6 @@ jobs:
- >-
pyspark-pandas-slow-connect
exclude:
- # Always run if pyspark == 'true', even infra-image is skip (such as
non-master job)
- # In practice, the build will run in individual PR, but not against
the individual commit
- # in Apache Spark repository.
- modules: ${{ fromJson(needs.precondition.outputs.required).pyspark
!= 'true' && 'pyspark-sql, pyspark-resource, pyspark-testing, pyspark-core,
pyspark-errors, pyspark-logger' }}
- modules: ${{ fromJson(needs.precondition.outputs.required).pyspark
!= 'true' && 'pyspark-mllib, pyspark-ml, pyspark-ml-connect' }}
- modules: ${{ fromJson(needs.precondition.outputs.required).pyspark
!= 'true' && 'pyspark-streaming, pyspark-structured-streaming,
pyspark-structured-streaming-connect' }}
@@ -670,9 +665,6 @@ jobs:
- modules: ${{
fromJson(needs.precondition.outputs.required).pyspark-connect-old-client !=
'true' && 'pyspark-connect-old-client'}}
# pyspark-install is very slow so we only run it when it's changed
or explicity requested
- modules: ${{
fromJson(needs.precondition.outputs.required).pyspark-install != 'true' &&
'pyspark-install' }}
- # Always run if pyspark-pandas == 'true', even infra-image is skip
(such as non-master job)
- # In practice, the build will run in individual PR, but not against
the individual commit
- # in Apache Spark repository.
- modules: ${{
fromJson(needs.precondition.outputs.required).pyspark-pandas != 'true' &&
'pyspark-pandas' }}
- modules: ${{
fromJson(needs.precondition.outputs.required).pyspark-pandas != 'true' &&
'pyspark-pandas-slow' }}
- modules: ${{
fromJson(needs.precondition.outputs.required).pyspark-pandas != 'true' &&
'pyspark-pandas-connect' }}
@@ -855,8 +847,7 @@ jobs:
sparkr:
needs: [precondition, infra-image, precompile]
- # always run if sparkr == 'true', even infra-image is skip (such as
non-master job)
- if: (!cancelled()) && fromJson(needs.precondition.outputs.required).sparkr
== 'true'
+ if: fromJson(needs.precondition.outputs.required).sparkr == 'true'
name: "Build modules: sparkr"
runs-on: ubuntu-latest
timeout-minutes: 120
@@ -1049,8 +1040,7 @@ jobs:
# Static analysis
lint:
needs: [precondition, infra-image]
- # always run if lint == 'true', even infra-image is skip (such as
non-master job)
- if: (!cancelled()) && fromJson(needs.precondition.outputs.required).lint
== 'true'
+ if: fromJson(needs.precondition.outputs.required).lint == 'true'
name: Linters, licenses, and dependencies
runs-on: ubuntu-latest
timeout-minutes: 120
@@ -1183,8 +1173,7 @@ jobs:
# Documentation build
docs:
needs: [precondition, infra-image]
- # always run if lint == 'true', even infra-image is skip (such as
non-master job)
- if: (!cancelled()) && fromJson(needs.precondition.outputs.required).docs
== 'true'
+ if: fromJson(needs.precondition.outputs.required).docs == 'true'
name: Documentation generation
runs-on: ubuntu-latest
timeout-minutes: 120
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
