This is an automated email from the ASF dual-hosted git repository.
dongjoon-hyun pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/spark-kubernetes-operator.git
The following commit(s) were added to refs/heads/main by this push:
new 56ff3e0 [SPARK-57153] Run CI in forked repositories and report build
status to PRs
56ff3e0 is described below
commit 56ff3e01f7159cbddac432e70a3dc6aed7ab6864
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Fri May 29 12:38:10 2026 -0700
[SPARK-57153] Run CI in forked repositories and report build status to PRs
### What changes were proposed in this pull request?
Make `build_and_test.yml` a reusable (`workflow_call`) workflow and add
Apache
Spark's fork-based CI status mechanism:
- `build_main.yml` — runs on `push` to all branches and calls
`build_and_test.yml`;
it also carries the `cancel-in-progress` concurrency group (moved out of
`build_and_test.yml`, since the triggers now live here).
- `notify_test_workflow.yml` — on `pull_request_target`, creates a `Build`
check
on the PR that points to the fork's run.
- `update_build_status.yml` — every 15 minutes, syncs the `Build` check
from the
fork's run.
- Add `images/workflow-enable-button.png` used by the notify message, and
repoint
the README badge to `build_main.yml`.
Flow:
```mermaid
flowchart TD
subgraph fork["Forked repo (contributor)"]
P([push to any branch]) --> BM["build_main.yml — job: Run"]
BM -->|workflow_call| BAT["build_and_test.yml
(reusable):<br/>License Check, build/test matrix, K8s and Helm tests"]
BAT --> CR["check runs:<br/>Run / License Check, ..."]
end
subgraph up["Upstream repo (apache/spark-kubernetes-operator)"]
PRT([pull_request_target]) --> N["notify_test_workflow.yml"]
SCH([schedule: every 15 min]) --> U["update_build_status.yml"]
N -->|create| B(["Build check on PR"])
U -->|"PATCH status / conclusion"| B
end
BM -.->|"① find run (id: build_main.yml)"| N
CR -.->|"② link check-run view"| N
BM -.->|"③ poll run status"| U
```
### Why are the changes needed?
Like the Apache Spark main repository, run the heavy test matrix in
contributors'
forks and mirror only a single `Build` status check upstream, instead of
running
the matrix in the upstream repo on every PR. This matches the Apache Spark
main
repository and is especially valuable here, where the matrix spans
multi-arch
build/test across JDK 21 / 25 / 26 plus dozens of Kubernetes integration
and Helm
jobs.
### Does this PR introduce _any_ user-facing change?
No. CI and documentation only.
### How was this patch tested?
Static verification of the cross-workflow wiring: `build_main.yml`'s `Run`
job and
`build_and_test.yml`'s `License Check` job produce the `Run / License Check`
check-run name that `notify_test_workflow.yml` probes for; the shared
`Build` check
name; and the `run_id` passed via the check output text from `notify` to
`update`.
All four workflow YAMLs parse cleanly. Full behavior takes effect only
after merge
to `main`, since the `pull_request_target` / `schedule` workflows and the
`main`-pinned image URL resolve from the default branch.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 4.8)
Closes #697 from dongjoon-hyun/SPARK-57153.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
---
.github/workflows/build_and_test.yml | 12 +-
.github/workflows/build_main.yml | 34 ++++
.../workflows/images/workflow-enable-button.png | Bin 0 -> 79807 bytes
.github/workflows/notify_test_workflow.yml | 172 +++++++++++++++++++++
.github/workflows/update_build_status.yml | 111 +++++++++++++
README.md | 2 +-
6 files changed, 319 insertions(+), 12 deletions(-)
diff --git a/.github/workflows/build_and_test.yml
b/.github/workflows/build_and_test.yml
index be4bf46..d75e552 100644
--- a/.github/workflows/build_and_test.yml
+++ b/.github/workflows/build_and_test.yml
@@ -1,17 +1,7 @@
name: Build and test
on:
- push:
- branches:
- - main
- pull_request:
- branches:
- - main
-
-# Cancel previous PR build and test
-concurrency:
- group: ${{ github.workflow }}-${{ github.event_name == 'pull_request' &&
github.event.number || github.sha }}
- cancel-in-progress: true
+ workflow_call:
jobs:
license-check:
diff --git a/.github/workflows/build_main.yml b/.github/workflows/build_main.yml
new file mode 100644
index 0000000..1c88907
--- /dev/null
+++ b/.github/workflows/build_main.yml
@@ -0,0 +1,34 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+name: "Build"
+
+on:
+ push:
+ branches: [ "**" ]
+
+# Cancel previous build on the same branch
+concurrency:
+ group: ${{ github.workflow }}-${{ github.ref }}
+ cancel-in-progress: true
+
+jobs:
+ call-build-and-test:
+ name: Run
+ uses: ./.github/workflows/build_and_test.yml
diff --git a/.github/workflows/images/workflow-enable-button.png
b/.github/workflows/images/workflow-enable-button.png
new file mode 100644
index 0000000..f7299f2
Binary files /dev/null and
b/.github/workflows/images/workflow-enable-button.png differ
diff --git a/.github/workflows/notify_test_workflow.yml
b/.github/workflows/notify_test_workflow.yml
new file mode 100644
index 0000000..03debf0
--- /dev/null
+++ b/.github/workflows/notify_test_workflow.yml
@@ -0,0 +1,172 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+# Intentionally has a general name.
+# because the test status check created in GitHub Actions
+# currently randomly picks any associated workflow.
+# So, the name was changed to make sense in that context too.
+# See also
https://github.community/t/specify-check-suite-when-creating-a-checkrun/118380/10
+name: On pull request update
+on:
+ pull_request_target:
+ types: [opened, reopened, synchronize]
+
+jobs:
+ notify:
+ name: Notify test workflow
+ runs-on: ubuntu-slim
+ permissions:
+ actions: read
+ checks: write
+ steps:
+ - name: "Notify test workflow"
+ uses: actions/github-script@v9
+ with:
+ github-token: ${{ secrets.GITHUB_TOKEN }}
+ script: |
+ const endpoint = 'GET
/repos/:owner/:repo/actions/workflows/:id/runs?&branch=:branch'
+ const check_run_endpoint = 'GET
/repos/:owner/:repo/commits/:ref/check-runs?per_page=100'
+
+ // TODO: Should use pull_request.user and
pull_request.user.repos_url?
+ // If a different person creates a commit to another forked repo,
+ // it wouldn't be able to detect.
+ const params = {
+ owner: context.payload.pull_request.head.repo.owner.login,
+ repo: context.payload.pull_request.head.repo.name,
+ id: 'build_main.yml',
+ branch: context.payload.pull_request.head.ref,
+ }
+ const check_run_params = {
+ owner: context.payload.pull_request.head.repo.owner.login,
+ repo: context.payload.pull_request.head.repo.name,
+ ref: context.payload.pull_request.head.ref,
+ }
+
+ console.log('Ref: ' + context.payload.pull_request.head.ref)
+ console.log('SHA: ' + context.payload.pull_request.head.sha)
+
+ // Wait 3 seconds to make sure the fork repository triggered a
workflow.
+ await new Promise(r => setTimeout(r, 3000))
+
+ let runs
+ try {
+ runs = await github.request(endpoint, params)
+ } catch (error) {
+ console.error(error)
+ // Assume that runs were not found.
+ }
+
+ const name = 'Build'
+ const head_sha = context.payload.pull_request.head.sha
+ let status = 'queued'
+
+ if (!runs || runs.data.workflow_runs.length === 0) {
+ status = 'completed'
+ const conclusion = 'action_required'
+
+ github.rest.checks.create({
+ owner: context.repo.owner,
+ repo: context.repo.repo,
+ name: name,
+ head_sha: head_sha,
+ status: status,
+ conclusion: conclusion,
+ output: {
+ title: 'Workflow run detection failed',
+ summary: `
+ Unable to detect the workflow run for testing the changes in your
PR.
+
+ 1. If you did not enable GitHub Actions in your forked repository,
please enable it by clicking the button as shown in the image below. See also
[Managing Github Actions Settings for a
repository](https://docs.github.com/en/repositories/managing-your-repositorys-settings-and-features/enabling-features-for-your-repository/managing-github-actions-settings-for-a-repository)
for more details.
+ 2. It is possible your branch is based on the old \`main\` branch
in Apache Spark K8s Operator, please sync your branch to the latest main
branch. For example as below:
+ \`\`\`bash
+ git fetch upstream
+ git rebase upstream/main
+ git push origin YOUR_BRANCH --force
+ \`\`\``,
+ images: [
+ {
+ alt: 'enabling workflows button',
+ image_url:
'https://raw.githubusercontent.com/apache/spark-kubernetes-operator/main/.github/workflows/images/workflow-enable-button.png'
+ }
+ ]
+ }
+ })
+ } else {
+ const run_id = runs.data.workflow_runs[0].id
+
+ if (runs.data.workflow_runs[0].head_sha !=
context.payload.pull_request.head.sha) {
+ throw new Error('There was a new unsynced commit pushed.
Please retrigger the workflow.');
+ }
+
+ // Here we get check run ID to provide Check run view instead of
Actions view, see also SPARK-37879.
+ let retryCount = 0;
+ let check_run_head;
+ while (retryCount < 3) {
+ const check_runs = await github.request(check_run_endpoint,
check_run_params);
+ check_run_head = check_runs.data.check_runs.find(r => r.name
=== "Run / License Check");
+ if (check_run_head) {
+ break;
+ }
+ retryCount++;
+ if (retryCount < 3) {
+ await new Promise(resolve => setTimeout(resolve, 3000));
+ }
+ }
+ if (!check_run_head) {
+ throw new Error('Failed to retrieve check_run_head after 3
attempts');
+ }
+
+ if (check_run_head.head_sha !=
context.payload.pull_request.head.sha) {
+ throw new Error('There was a new unsynced commit pushed.
Please retrigger the workflow.');
+ }
+
+ const check_run_url = 'https://github.com/'
+ + context.payload.pull_request.head.repo.full_name
+ + '/runs/'
+ + check_run_head.id
+ console.log('Check run URL: ' + check_run_url)
+
+ const actions_url = 'https://github.com/'
+ + context.payload.pull_request.head.repo.full_name
+ + '/actions/runs/'
+ + run_id
+ console.log('Actions URL: ' + actions_url)
+
+ github.rest.checks.create({
+ owner: context.repo.owner,
+ repo: context.repo.repo,
+ name: name,
+ head_sha: head_sha,
+ status: status,
+ output: {
+ title: 'Test results',
+ summary: '[See test results](' + check_run_url + ')\n\n'
+ + 'If the tests fail for reasons unrelated to this pull
request, '
+ + 'please rerun the workflow in your forked repository.\n'
+ + 'If the failures are related to this pull request, '
+ + 'please investigate them and push follow-up changes.',
+ text: JSON.stringify({
+ owner: context.payload.pull_request.head.repo.owner.login,
+ repo: context.payload.pull_request.head.repo.name,
+ run_id: run_id
+ })
+ },
+ details_url: actions_url,
+ })
+ }
diff --git a/.github/workflows/update_build_status.yml
b/.github/workflows/update_build_status.yml
new file mode 100644
index 0000000..26ab78f
--- /dev/null
+++ b/.github/workflows/update_build_status.yml
@@ -0,0 +1,111 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+name: Update build status workflow
+
+on:
+ schedule:
+ - cron: "*/15 * * * *"
+
+jobs:
+ update:
+ name: Update build status
+ runs-on: ubuntu-slim
+ permissions:
+ actions: read
+ checks: write
+ steps:
+ - name: "Update build status"
+ uses: actions/github-script@v9
+ with:
+ github-token: ${{ secrets.GITHUB_TOKEN }}
+ script: |
+ const endpoint = 'GET /repos/:owner/:repo/pulls?state=:state'
+ const params = {
+ owner: context.repo.owner,
+ repo: context.repo.repo,
+ state: 'open'
+ }
+
+ // See
https://docs.github.com/en/graphql/reference/enums#mergestatestatus
+ const maybeReady = ['behind', 'clean', 'draft', 'has_hooks',
'unknown', 'unstable'];
+
+ // Iterate open PRs
+ for await (const prs of github.paginate.iterator(endpoint,params))
{
+ // Each page
+ for await (const pr of prs.data) {
+ console.log('SHA: ' + pr.head.sha)
+ console.log(' Mergeable status: ' + pr.mergeable_state)
+ if (pr.mergeable_state == null ||
maybeReady.includes(pr.mergeable_state)) {
+ const checkRuns = await github.request('GET
/repos/{owner}/{repo}/commits/{ref}/check-runs', {
+ owner: context.repo.owner,
+ repo: context.repo.repo,
+ ref: pr.head.sha
+ })
+
+ // Iterator GitHub Checks in the PR
+ for await (const cr of checkRuns.data.check_runs) {
+ if (cr.name == 'Build' && cr.conclusion !=
"action_required") {
+ // text contains parameters to make request in JSON.
+ const params = JSON.parse(cr.output.text)
+
+ // Get the workflow run in the forked repository
+ let run
+ try {
+ run = await github.request('GET
/repos/{owner}/{repo}/actions/runs/{run_id}', params)
+ } catch (error) {
+ console.error(error)
+ // Run not found. This can happen when the PR author
removes GitHub Actions runs or
+ // disables GitHub Actions.
+ continue
+ }
+
+ // Keep syncing the status of the checks
+ if (run.data.status == 'completed') {
+ console.log(' Run ' + cr.id + ': set status (' +
run.data.status + ') and conclusion (' + run.data.conclusion + ')')
+ const response = await github.request('PATCH
/repos/{owner}/{repo}/check-runs/{check_run_id}', {
+ owner: context.repo.owner,
+ repo: context.repo.repo,
+ check_run_id: cr.id,
+ output: cr.output,
+ status: run.data.status,
+ conclusion: run.data.conclusion,
+ details_url: run.data.details_url
+ })
+ } else {
+ // PATCH /check-runs accepts only queued | in_progress
| completed,
+ // but workflow_run may also report requested /
waiting / pending.
+ const status = run.data.status == 'in_progress' ?
'in_progress' : 'queued'
+ console.log(' Run ' + cr.id + ': set status (' +
run.data.status + ' -> ' + status + ')')
+ const response = await github.request('PATCH
/repos/{owner}/{repo}/check-runs/{check_run_id}', {
+ owner: context.repo.owner,
+ repo: context.repo.repo,
+ check_run_id: cr.id,
+ output: cr.output,
+ status: status,
+ details_url: run.data.details_url
+ })
+ }
+
+ break
+ }
+ }
+ }
+ }
+ }
diff --git a/README.md b/README.md
index 5e101bb..741c444 100644
--- a/README.md
+++ b/README.md
@@ -3,7 +3,7 @@
[](https://opensource.org/licenses/Apache-2.0)
[](https://github.com/apache/spark-kubernetes-operator/releases)
[](https://artifacthub.io/packages/search?repo=spark-kubernetes-operator)
-[](https://github.com/apache/spark-kubernetes-operator/actions/workflows/build_and_test.yml)
+[](https://github.com/apache/spark-kubernetes-operator/actions/workflows/build_main.yml)
[](https://adoptium.net/temurin/releases/?version=21)
[](https://img.shields.io/github/repo-size/apache/spark-kubernetes-operator)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
