This is an automated email from the ASF dual-hosted git repository.
gaogaotiantian pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 28c232f295ed [SPARK-57483][INFRA] Move scheduled CIs for 4.0 to
branch-4.0
28c232f295ed is described below
commit 28c232f295edf06836667121ad859a730f8163d4
Author: Tian Gao <[email protected]>
AuthorDate: Wed Jun 17 11:13:19 2026 -0700
[SPARK-57483][INFRA] Move scheduled CIs for 4.0 to branch-4.0
### What changes were proposed in this pull request?
Add a unified entry for all scheduled CIs for branch-4.0
(`branch40_scheduler.yml`). It uses `gh workflow` to trigger the self-contained
build workflows on `branch-4.0`, and removes the per-build
`build_branch40_*.yml` files from `master`.
This follows the same approach as SPARK-56990 (#56046) for `branch-4.x` and
SPARK-57327 (#56379) for `branch-4.1`. SPARK-57398 (#56461) already laid the
ground on `branch-4.0` by making the build workflows self-contained and
dispatchable, so this PR only needs to change the `master` scheduled tasks.
The scheduler triggers the following targets on `branch-4.0` (also exposed
via `workflow_dispatch`): `build_java17`, `build_java21`, `build_maven`,
`build_maven_java21`, `build_non_ansi`, `build_python_3.11`,
`build_python_pypy3.10`. Note this differs from the `branch-4.1` set: there is
no `python_3.14` build on `branch-4.0`. The cron schedule preserves the
original times and `*/4` frequency from the `build_branch40_*.yml` files (the
Java 17 and Python 3.11 builds share `0 12`, and th [...]
`README.md` is updated so the branch-4.0 badges point at the self-contained
workflows filtered by `?branch=branch-4.0`.
### Why are the changes needed?
This is part of decoupling our CIs. All `branch-4.0` related CIs should
only rely on files on `branch-4.0`, with the exception of this new scheduler
file which is needed on `master` to trigger scheduled tasks (scheduled
workflows only fire from the default branch).
### Does this PR introduce _any_ user-facing change?
No. CI only.
### How was this patch tested?
These workflows can be triggered manually via `workflow_dispatch` once
merged.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 4.8)
This pull request and its description were written by Isaac.
Closes #56530 from gaogaotiantian/decouple-branch40-scheduler.
Authored-by: Tian Gao <[email protected]>
Signed-off-by: Tian Gao <[email protected]>
---
.github/workflows/branch40_scheduler.yml | 103 +++++++++++++++++++++
.github/workflows/build_branch40.yml | 53 -----------
.github/workflows/build_branch40_java21.yml | 57 ------------
.github/workflows/build_branch40_maven.yml | 35 -------
.github/workflows/build_branch40_maven_java21.yml | 36 -------
.github/workflows/build_branch40_non_ansi.yml | 54 -----------
.github/workflows/build_branch40_python.yml | 47 ----------
.../workflows/build_branch40_python_pypy3.10.yml | 47 ----------
README.md | 15 +--
9 files changed, 111 insertions(+), 336 deletions(-)
diff --git a/.github/workflows/branch40_scheduler.yml
b/.github/workflows/branch40_scheduler.yml
new file mode 100644
index 000000000000..b6fb0e9af5fc
--- /dev/null
+++ b/.github/workflows/branch40_scheduler.yml
@@ -0,0 +1,103 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+name: "Branch-4.0 CI Scheduler"
+
+on:
+ schedule:
+ - cron: '0 2 */4 * *'
+ - cron: '0 5 */4 * *'
+ - cron: '0 12 */4 * *'
+ - cron: '0 14 */4 * *'
+ - cron: '0 16 */4 * *'
+ workflow_dispatch:
+ inputs:
+ target:
+ description: Target workflow to run
+ required: true
+ type: choice
+ default: all
+ options:
+ - all
+ - maven
+ - maven_java21
+ - java17
+ - java21
+ - non_ansi
+ - python_3.11
+ - python_pypy3.10
+
+jobs:
+ schedule:
+ if: github.repository == 'apache/spark'
+ runs-on: ubuntu-latest
+ env:
+ GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+ permissions:
+ actions: write
+ steps:
+ - name: Maven Build
+ if: >-
+ github.event.schedule == '0 14 */4 * *' ||
+ (github.event_name == 'workflow_dispatch' &&
+ (inputs.target == 'maven' || inputs.target == 'all'))
+ run: |
+ gh workflow run build_maven.yml --repo ${{ github.repository }}
--ref branch-4.0
+ - name: Maven Java 21 Build
+ if: >-
+ github.event.schedule == '0 14 */4 * *' ||
+ (github.event_name == 'workflow_dispatch' &&
+ (inputs.target == 'maven_java21' || inputs.target == 'all'))
+ run: |
+ gh workflow run build_maven_java21.yml --repo ${{ github.repository
}} --ref branch-4.0
+ - name: Java 17 Build
+ if: >-
+ github.event.schedule == '0 12 */4 * *' ||
+ (github.event_name == 'workflow_dispatch' &&
+ (inputs.target == 'java17' || inputs.target == 'all'))
+ run: |
+ gh workflow run build_java17.yml --repo ${{ github.repository }}
--ref branch-4.0
+ - name: Java 21 Build
+ if: >-
+ github.event.schedule == '0 5 */4 * *' ||
+ (github.event_name == 'workflow_dispatch' &&
+ (inputs.target == 'java21' || inputs.target == 'all'))
+ run: |
+ gh workflow run build_java21.yml --repo ${{ github.repository }}
--ref branch-4.0
+ - name: Non-ANSI Build
+ if: >-
+ github.event.schedule == '0 2 */4 * *' ||
+ (github.event_name == 'workflow_dispatch' &&
+ (inputs.target == 'non_ansi' || inputs.target == 'all'))
+ run: |
+ gh workflow run build_non_ansi.yml --repo ${{ github.repository }}
--ref branch-4.0
+ - name: Python 3.11 Build
+ if: >-
+ github.event.schedule == '0 12 */4 * *' ||
+ (github.event_name == 'workflow_dispatch' &&
+ (inputs.target == 'python_3.11' || inputs.target == 'all'))
+ run: |
+ gh workflow run build_python_3.11.yml --repo ${{ github.repository
}} --ref branch-4.0
+ - name: Python PyPy 3.10 Build
+ if: >-
+ github.event.schedule == '0 16 */4 * *' ||
+ (github.event_name == 'workflow_dispatch' &&
+ (inputs.target == 'python_pypy3.10' || inputs.target == 'all'))
+ run: |
+ gh workflow run build_python_pypy3.10.yml --repo ${{
github.repository }} --ref branch-4.0
diff --git a/.github/workflows/build_branch40.yml
b/.github/workflows/build_branch40.yml
deleted file mode 100644
index a4e1c68b9c9d..000000000000
--- a/.github/workflows/build_branch40.yml
+++ /dev/null
@@ -1,53 +0,0 @@
-#
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied. See the License for the
-# specific language governing permissions and limitations
-# under the License.
-#
-
-name: "Build (branch-4.0, Scala 2.13, Hadoop 3, JDK 17)"
-
-on:
- schedule:
- - cron: '0 12 */4 * *'
- workflow_dispatch:
-
-jobs:
- run-build:
- permissions:
- packages: write
- name: Run
- uses: ./.github/workflows/build_and_test.yml
- if: github.repository == 'apache/spark'
- with:
- java: 17
- branch: branch-4.0
- hadoop: hadoop3
- envs: >-
- {
- "SCALA_PROFILE": "scala2.13",
- "PYSPARK_IMAGE_TO_TEST": "",
- "PYTHON_TO_TEST": "",
- "ORACLE_DOCKER_IMAGE_NAME": "gvenzl/oracle-free:23.7-slim"
- }
- jobs: >-
- {
- "build": "true",
- "sparkr": "true",
- "tpcds-1g": "true",
- "docker-integration-tests": "true",
- "k8s-integration-tests": "true",
- "lint" : "true"
- }
diff --git a/.github/workflows/build_branch40_java21.yml
b/.github/workflows/build_branch40_java21.yml
deleted file mode 100644
index bfa1a8044789..000000000000
--- a/.github/workflows/build_branch40_java21.yml
+++ /dev/null
@@ -1,57 +0,0 @@
-#
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied. See the License for the
-# specific language governing permissions and limitations
-# under the License.
-#
-
-name: "Build (branch-4.0, Scala 2.13, Hadoop 3, JDK 21)"
-
-on:
- schedule:
- - cron: '0 5 */4 * *'
- workflow_dispatch:
-
-jobs:
- run-build:
- permissions:
- packages: write
- name: Run
- uses: ./.github/workflows/build_and_test.yml
- if: github.repository == 'apache/spark'
- with:
- java: 21
- branch: branch-4.0
- hadoop: hadoop3
- envs: >-
- {
- "PYSPARK_IMAGE_TO_TEST": "python-311",
- "PYTHON_TO_TEST": "python3.11",
- "SKIP_MIMA": "true",
- "SKIP_UNIDOC": "true",
- "DEDICATED_JVM_SBT_TESTS":
"org.apache.spark.sql.execution.datasources.parquet.ParquetFileFormatV1Suite,org.apache.spark.sql.execution.datasources.parquet.ParquetFileFormatV2Suite,org.apache.spark.sql.execution.datasources.orc.OrcSourceV1Suite,org.apache.spark.sql.execution.datasources.orc.OrcSourceV2Suite"
- }
- jobs: >-
- {
- "build": "true",
- "pyspark": "true",
- "sparkr": "true",
- "tpcds-1g": "true",
- "docker-integration-tests": "true",
- "yarn": "true",
- "k8s-integration-tests": "true",
- "buf": "true",
- "ui": "true"
- }
diff --git a/.github/workflows/build_branch40_maven.yml
b/.github/workflows/build_branch40_maven.yml
deleted file mode 100644
index 30d55bac7ef2..000000000000
--- a/.github/workflows/build_branch40_maven.yml
+++ /dev/null
@@ -1,35 +0,0 @@
-#
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied. See the License for the
-# specific language governing permissions and limitations
-# under the License.
-#
-
-name: "Build / Maven (branch-4.0, Scala 2.13, Hadoop 3, JDK 17)"
-
-on:
- schedule:
- - cron: '0 14 */4 * *'
- workflow_dispatch:
-
-jobs:
- run-build:
- permissions:
- packages: write
- name: Run
- uses: ./.github/workflows/maven_test.yml
- if: github.repository == 'apache/spark'
- with:
- branch: branch-4.0
diff --git a/.github/workflows/build_branch40_maven_java21.yml
b/.github/workflows/build_branch40_maven_java21.yml
deleted file mode 100644
index b63b613294e5..000000000000
--- a/.github/workflows/build_branch40_maven_java21.yml
+++ /dev/null
@@ -1,36 +0,0 @@
-#
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied. See the License for the
-# specific language governing permissions and limitations
-# under the License.
-#
-
-name: "Build / Maven (branch-4.0, Scala 2.13, Hadoop 3, JDK 21)"
-
-on:
- schedule:
- - cron: '0 14 */4 * *'
- workflow_dispatch:
-
-jobs:
- run-build:
- permissions:
- packages: write
- name: Run
- uses: ./.github/workflows/maven_test.yml
- if: github.repository == 'apache/spark'
- with:
- branch: branch-4.0
- java: 21
diff --git a/.github/workflows/build_branch40_non_ansi.yml
b/.github/workflows/build_branch40_non_ansi.yml
deleted file mode 100644
index e5fe4e404485..000000000000
--- a/.github/workflows/build_branch40_non_ansi.yml
+++ /dev/null
@@ -1,54 +0,0 @@
-#
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied. See the License for the
-# specific language governing permissions and limitations
-# under the License.
-#
-
-name: "Build / Non-ANSI (branch-4.0, Hadoop 3, JDK 17, Scala 2.13)"
-
-on:
- schedule:
- - cron: '0 2 */4 * *'
- workflow_dispatch:
-
-jobs:
- run-build:
- permissions:
- packages: write
- name: Run
- uses: ./.github/workflows/build_and_test.yml
- if: github.repository == 'apache/spark'
- with:
- java: 17
- branch: branch-4.0
- hadoop: hadoop3
- envs: >-
- {
- "PYSPARK_IMAGE_TO_TEST": "python-311",
- "PYTHON_TO_TEST": "python3.11",
- "SPARK_ANSI_SQL_MODE": "false",
- }
- jobs: >-
- {
- "build": "true",
- "build-core-utils": "false",
- "docs": "true",
- "pyspark": "true",
- "sparkr": "true",
- "tpcds-1g": "true",
- "docker-integration-tests": "true",
- "yarn": "true"
- }
diff --git a/.github/workflows/build_branch40_python.yml
b/.github/workflows/build_branch40_python.yml
deleted file mode 100644
index d0245dd5ec77..000000000000
--- a/.github/workflows/build_branch40_python.yml
+++ /dev/null
@@ -1,47 +0,0 @@
-#
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied. See the License for the
-# specific language governing permissions and limitations
-# under the License.
-#
-
-name: "Build / Python-only (branch-4.0)"
-
-on:
- schedule:
- - cron: '0 12 */4 * *'
- workflow_dispatch:
-
-jobs:
- run-build:
- permissions:
- packages: write
- name: Run
- uses: ./.github/workflows/build_and_test.yml
- if: github.repository == 'apache/spark'
- with:
- java: 17
- branch: branch-4.0
- hadoop: hadoop3
- envs: >-
- {
- "PYSPARK_IMAGE_TO_TEST": "python-311",
- "PYTHON_TO_TEST": "python3.11"
- }
- jobs: >-
- {
- "pyspark": "true",
- "pyspark-pandas": "true"
- }
diff --git a/.github/workflows/build_branch40_python_pypy3.10.yml
b/.github/workflows/build_branch40_python_pypy3.10.yml
deleted file mode 100644
index 43d515df7cac..000000000000
--- a/.github/workflows/build_branch40_python_pypy3.10.yml
+++ /dev/null
@@ -1,47 +0,0 @@
-#
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied. See the License for the
-# specific language governing permissions and limitations
-# under the License.
-#
-
-name: "Build / Python-only (branch-4.0, PyPy 3.10)"
-
-on:
- schedule:
- - cron: '0 16 */4 * *'
- workflow_dispatch:
-
-jobs:
- run-build:
- permissions:
- packages: write
- name: Run
- uses: ./.github/workflows/build_and_test.yml
- if: github.repository == 'apache/spark'
- with:
- java: 17
- branch: branch-4.0
- hadoop: hadoop3
- envs: >-
- {
- "PYSPARK_IMAGE_TO_TEST": "pypy-310",
- "PYTHON_TO_TEST": "pypy3"
- }
- jobs: >-
- {
- "pyspark": "true",
- "pyspark-pandas": "true"
- }
diff --git a/README.md b/README.md
index 9881c5636896..0a998223b6d1 100644
--- a/README.md
+++ b/README.md
@@ -80,13 +80,14 @@ This README file only contains basic setup instructions.
| | [](https://github.com/apache/spark/actions/workflows/build_python_3.11.yml?query=branch%3Abranch-4.1)
|
| | [](https://github.com/apache/spark/actions/workflows/build_python_3.14.yml?query=branch%3Abranch-4.1)
|
| | [](https://github.com/apache/spark/actions/workflows/build_python_pypy3.10.yml?query=branch%3Abranch-4.1)
|
-| branch-4.0 | [](https://github.com/apache/spark/actions/workflows/build_branch40.yml)
|
-| | [](https://github.com/apache/spark/actions/workflows/build_branch40_java21.yml)
|
-| | [](https://github.com/apache/spark/actions/workflows/build_branch40_non_ansi.yml)
|
-| | [](https://github.com/apache/spark/actions/workflows/build_branch40_maven.yml)
|
-| | [](https://github.com/apache/spark/actions/workflows/build_branch40_maven_java21.yml)
|
-| | [](https://github.com/apache/spark/actions/workflows/build_branch40_python.yml)
|
-| | [](https://github.com/apache/spark/actions/workflows/build_branch40_python_pypy3.10.yml)
|
+| branch-4.0 | [](https://github.com/apache/spark/actions/workflows/build_main.yml?query=branch%3Abranch-4.0)
|
+| | [](https://github.com/apache/spark/actions/workflows/build_java17.yml?query=branch%3Abranch-4.0)
|
+| | [](https://github.com/apache/spark/actions/workflows/build_java21.yml?query=branch%3Abranch-4.0)
|
+| | [](https://github.com/apache/spark/actions/workflows/build_non_ansi.yml?query=branch%3Abranch-4.0)
|
+| | [](https://github.com/apache/spark/actions/workflows/build_maven.yml?query=branch%3Abranch-4.0)
|
+| | [](https://github.com/apache/spark/actions/workflows/build_maven_java21.yml?query=branch%3Abranch-4.0)
|
+| | [](https://github.com/apache/spark/actions/workflows/build_python_3.11.yml?query=branch%3Abranch-4.0)
|
+| | [](https://github.com/apache/spark/actions/workflows/build_python_pypy3.10.yml?query=branch%3Abranch-4.0)
|
| branch-3.5 | [](https://github.com/apache/spark/actions/workflows/build_branch35.yml)
|
| | [](https://github.com/apache/spark/actions/workflows/build_branch35_python.yml)
|
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
