This is an automated email from the ASF dual-hosted git repository.
zhengruifeng pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 3ce5de6baa38 [SPARK-58285][INFRA] Remove stale container CI dependency
bypasses
3ce5de6baa38 is described below
commit 3ce5de6baa381f1ee07b3d334a8c5ec3e06dbc25
Author: Ruifeng Zheng <[email protected]>
AuthorDate: Thu Jul 23 20:40:01 2026 +0800
[SPARK-58285][INFRA] Remove stale container CI dependency bypasses
### What changes were proposed in this pull request?
Removes the cancelled-status dependency bypasses from the PySpark, SparkR,
lint, and documentation jobs. Replaces the PySpark matrix's obsolete non-master
fallback comments with descriptions of the entries selected by each change gate.
### Why are the changes needed?
The bypasses were introduced when non-master jobs used static fallback
container images. These jobs now use per-run images produced by infra-image.
When that build fails, the bypass still schedules the entire PySpark matrix and
the SparkR, lint, and documentation jobs; they then fail minutes later while
trying to start from images that were never pushed. No dependent container job
can run successfully in that state, so GitHub Actions should block them until
infra-image succeeds.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Ran git diff --check. No test suite was run because this is a GitHub
Actions job-condition and comment change.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Codex (GPT-5)
Closes #57457 from zhengruifeng/remove-pyspark-needs-bypass-dev4.
Authored-by: Ruifeng Zheng <[email protected]>
Signed-off-by: Ruifeng Zheng <[email protected]>
---
.github/workflows/build_and_test.yml | 23 ++++++-----------------
1 file changed, 6 insertions(+), 17 deletions(-)
diff --git a/.github/workflows/build_and_test.yml
b/.github/workflows/build_and_test.yml
index 0bcda10e3b6c..342e4339bde2 100644
--- a/.github/workflows/build_and_test.yml
+++ b/.github/workflows/build_and_test.yml
@@ -325,9 +325,6 @@ jobs:
excluded-tags:
org.apache.spark.tags.ExtendedSQLTest,org.apache.spark.tags.SlowSQLTest
comment: "- other tests"
exclude:
- # Always run if yarn == 'true', even infra-image is skip (such as
non-master job)
- # In practice, the build will run in individual PR, but not against
the individual commit
- # in Apache Spark repository.
- modules: ${{ fromJson(needs.precondition.outputs.required).yarn !=
'true' && 'yarn' }}
# Skip the core/utils group when a PR doesn't touch those modules
(e.g. SQL-only or
# PySpark-only changes). The precondition is opt-out: omitted means
run (so periodic
@@ -622,8 +619,9 @@ jobs:
pyspark:
needs: [precondition, infra-image, precompile]
- # always run if pyspark == 'true', even infra-image is skip (such as
non-master job)
- if: (!cancelled()) &&
(fromJson(needs.precondition.outputs.required).pyspark == 'true' ||
fromJson(needs.precondition.outputs.required).pyspark-pandas == 'true')
+ if: >-
+ fromJson(needs.precondition.outputs.required).pyspark == 'true' ||
+ fromJson(needs.precondition.outputs.required).pyspark-pandas == 'true'
name: "Build modules: ${{ matrix.modules }}"
runs-on: ubuntu-latest
timeout-minutes: 120
@@ -660,9 +658,6 @@ jobs:
- >-
pyspark-pandas-slow-connect
exclude:
- # Always run if pyspark == 'true', even infra-image is skip (such as
non-master job)
- # In practice, the build will run in individual PR, but not against
the individual commit
- # in Apache Spark repository.
- modules: ${{ fromJson(needs.precondition.outputs.required).pyspark
!= 'true' && 'pyspark-sql, pyspark-resource, pyspark-testing, pyspark-core,
pyspark-errors, pyspark-logger' }}
- modules: ${{ fromJson(needs.precondition.outputs.required).pyspark
!= 'true' && 'pyspark-mllib, pyspark-ml, pyspark-ml-connect' }}
- modules: ${{ fromJson(needs.precondition.outputs.required).pyspark
!= 'true' && 'pyspark-streaming, pyspark-structured-streaming,
pyspark-structured-streaming-connect' }}
@@ -670,9 +665,6 @@ jobs:
- modules: ${{
fromJson(needs.precondition.outputs.required).pyspark-connect-old-client !=
'true' && 'pyspark-connect-old-client'}}
# pyspark-install is very slow so we only run it when it's changed
or explicity requested
- modules: ${{
fromJson(needs.precondition.outputs.required).pyspark-install != 'true' &&
'pyspark-install' }}
- # Always run if pyspark-pandas == 'true', even infra-image is skip
(such as non-master job)
- # In practice, the build will run in individual PR, but not against
the individual commit
- # in Apache Spark repository.
- modules: ${{
fromJson(needs.precondition.outputs.required).pyspark-pandas != 'true' &&
'pyspark-pandas' }}
- modules: ${{
fromJson(needs.precondition.outputs.required).pyspark-pandas != 'true' &&
'pyspark-pandas-slow' }}
- modules: ${{
fromJson(needs.precondition.outputs.required).pyspark-pandas != 'true' &&
'pyspark-pandas-connect' }}
@@ -855,8 +847,7 @@ jobs:
sparkr:
needs: [precondition, infra-image, precompile]
- # always run if sparkr == 'true', even infra-image is skip (such as
non-master job)
- if: (!cancelled()) && fromJson(needs.precondition.outputs.required).sparkr
== 'true'
+ if: fromJson(needs.precondition.outputs.required).sparkr == 'true'
name: "Build modules: sparkr"
runs-on: ubuntu-latest
timeout-minutes: 120
@@ -1000,8 +991,7 @@ jobs:
# Static analysis
lint:
needs: [precondition, infra-image]
- # always run if lint == 'true', even infra-image is skip (such as
non-master job)
- if: (!cancelled()) && fromJson(needs.precondition.outputs.required).lint
== 'true'
+ if: fromJson(needs.precondition.outputs.required).lint == 'true'
name: Linters, licenses, and dependencies
runs-on: ubuntu-latest
timeout-minutes: 120
@@ -1143,8 +1133,7 @@ jobs:
# Documentation build
docs:
needs: [precondition, infra-image]
- # always run if lint == 'true', even infra-image is skip (such as
non-master job)
- if: (!cancelled()) && fromJson(needs.precondition.outputs.required).docs
== 'true'
+ if: fromJson(needs.precondition.outputs.required).docs == 'true'
name: Documentation generation
runs-on: ubuntu-latest
timeout-minutes: 120
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
