This is an automated email from the ASF dual-hosted git repository.
nchammas pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new c6c90769867d [SPARK-60578][INFRA] Migrate build status workflow off of
`pull_request_target` trigger
c6c90769867d is described below
commit c6c90769867dce1eca14643a60cbcbf89f8d2286
Author: Nicholas Chammas <[email protected]>
AuthorDate: Tue Nov 3 17:05:59 2026 -0500
[SPARK-60578][INFRA] Migrate build status workflow off of
`pull_request_target` trigger
### What changes were proposed in this pull request?
Remove `notify_test_workflow.yml` and rely on the existing
`update_build_status.yml` to report fork build status on PRs. Update the
comments accordingly.
This builds on
[apache/spark#58579](https://github.com/apache/spark/pull/58579) (SPARK-59293),
which fixes errors that abort the scheduled scan.
### Why are the changes needed?
ASF is disabling the `pull_request_target` trigger in [November 2026][asf],
ahead of GitHub's [December 2 deadline][github]. This is our only workflow
using it.
The scheduled updater already creates missing `Build` checks (SPARK-58660),
so we don't need another workflow to replace the notifier. Fork builds still
run on push, as usual; this PR only changes how their status gets reported.
We should backport this PR to active maintenance branches that accept PRs,
since `pull_request_target` uses the base branch's workflow.
[asf]: https://infra.apache.org/github-actions-policy.html
[github]:
https://docs.github.com/en/actions/reference/security/securely-using-pull_request_target#default-policy-for-pull_request_target
I intentionally kept this PR simple as I _really_ don't like that we have
to maintain so much GitHub Actions-specific JavaScript just to manage our
fork-based contribution workflow.
In other words, in the interest of simplicity I intentionally left out:
- Any mechanism to update PRs "live".
- Testing infrastructure for these workflows and the JavaScript they
contain.
This infrastructure is not in the Spark project's area of expertise. It
really feels like something we should be able to import and use from somewhere.
Until then, I think we should strive to keep these workflows as simple as
possible.
### Does this PR introduce _any_ user-facing change?
No, nothing for end users.
For contributors here on GitHub:
- The "Notify test workflow" check goes away.
- New `Build` checks appear as `queued` on the first scheduled pass and
pick up the fork result on the next. The schedule is every 15 minutes, but
GitHub can delay or drop runs.
- If you push multiple updates to a PR between scans, only the latest
commit gets a `Build` check.
- The results link opens the fork's Actions run instead of its `Run / Check
changes` check.
### How was this patch tested?
Checked locally with actionlint, YAML parsing, and git diff --check.
`update_build_status.yml` changes only comments, so the scheduled updater,
which runs from master, already behaves as it will after this PR. In [run
40517915808](https://github.com/apache/spark/actions/runs/40517915808) it
created Build checks for PRs that had none (#54953, #59831), and later
scheduled runs synced them (#54953 is now success).
Additionally, an agent replayed the unmodified updater against 150 open PRs
with their Build checks hidden and writes intercepted: all 150 resolve to the
same fork run, and the 36 action_required PRs would get identical checks. All
708 open master PRs currently have exactly one Build check, and 706 match their
fork run.
This PR can't fully test itself because `pull_request_target` runs the base
branch's copy of the notifier. I will run additional tests post merge.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Copilot SDK in VS Code (version unavailable)
Closes #59829 from nchammas/pr-target-actions-policy.
Authored-by: Nicholas Chammas <[email protected]>
Signed-off-by: Nicholas Chammas <[email protected]>
(cherry picked from commit 6a02fc4e8e5c9c34cfdf01b31fefcf84589a2298)
Signed-off-by: Nicholas Chammas <[email protected]>
---
.github/workflows/notify_test_workflow.yml | 198 -----------------------------
.github/workflows/update_build_status.yml | 67 +++++-----
2 files changed, 36 insertions(+), 229 deletions(-)
diff --git a/.github/workflows/notify_test_workflow.yml
b/.github/workflows/notify_test_workflow.yml
deleted file mode 100644
index 1ca9e14c408b..000000000000
--- a/.github/workflows/notify_test_workflow.yml
+++ /dev/null
@@ -1,198 +0,0 @@
-#
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied. See the License for the
-# specific language governing permissions and limitations
-# under the License.
-#
-
-# Intentionally has a general name.
-# because the test status check created in GitHub Actions
-# currently randomly picks any associated workflow.
-# So, the name was changed to make sense in that context too.
-# See also
https://github.community/t/specify-check-suite-when-creating-a-checkrun/118380/10
-name: On pull request update
-on:
- pull_request_target:
- types: [opened, reopened, synchronize]
-
-jobs:
- notify:
- name: Notify test workflow
- runs-on: ubuntu-latest
- permissions:
- actions: read
- checks: write
- steps:
- - name: "Notify test workflow"
- uses: actions/github-script@v8
- with:
- github-token: ${{ secrets.GITHUB_TOKEN }}
- script: |
- const endpoint = 'GET
/repos/:owner/:repo/actions/workflows/:id/runs?&branch=:branch'
- const check_run_endpoint = 'GET
/repos/:owner/:repo/commits/:ref/check-runs?per_page=100'
-
- // TODO: Should use pull_request.user and
pull_request.user.repos_url?
- // If a different person creates a commit to another forked repo,
- // it wouldn't be able to detect.
- const params = {
- owner: context.payload.pull_request.head.repo.owner.login,
- repo: context.payload.pull_request.head.repo.name,
- id: 'build_main.yml',
- branch: context.payload.pull_request.head.ref,
- }
- const check_run_params = {
- owner: context.payload.pull_request.head.repo.owner.login,
- repo: context.payload.pull_request.head.repo.name,
- ref: context.payload.pull_request.head.ref,
- }
-
- console.log('Ref: ' + context.payload.pull_request.head.ref)
- console.log('SHA: ' + context.payload.pull_request.head.sha)
-
- const name = 'Build'
- const head_sha = context.payload.pull_request.head.sha
- let status = 'queued'
-
- // Wait 3 seconds to make sure the fork repository triggered a
workflow.
- await new Promise(r => setTimeout(r, 3000))
-
- // The workflow run for this exact SHA may not be registered yet,
and the
- // listing endpoint orders by most recent, so blindly taking the
first run
- // can return a stale run from a previous push. Re-query a few
times looking
- // for the run whose head_sha matches this PR's head SHA before
giving up.
- let matched_run
- let any_runs = false
- let retryCount = 0
- while (retryCount < 3) {
- let runs
- try {
- runs = await github.request(endpoint, params)
- } catch (error) {
- console.error(error)
- // Assume that runs were not found.
- }
- if (runs && runs.data.workflow_runs.length > 0) {
- any_runs = true
- matched_run = runs.data.workflow_runs.find(r => r.head_sha ==
head_sha)
- if (matched_run) {
- break
- }
- }
- retryCount++
- if (retryCount < 3) {
- await new Promise(resolve => setTimeout(resolve, 3000))
- }
- }
-
- // If we saw runs but none matched the PR head SHA, a newer commit
was pushed
- // and a fresh notify run will handle it; nothing useful to report
for this SHA.
- if (any_runs && !matched_run) {
- throw new Error('There was a new unsynced commit pushed. Please
retrigger the workflow.');
- }
-
- if (!matched_run) {
- status = 'completed'
- const conclusion = 'action_required'
-
- github.rest.checks.create({
- owner: context.repo.owner,
- repo: context.repo.repo,
- name: name,
- head_sha: head_sha,
- status: status,
- conclusion: conclusion,
- output: {
- title: 'Workflow run detection failed',
- summary: `
- Unable to detect the workflow run for testing the changes in your
PR.
-
- 1. If you did not enable GitHub Actions in your forked repository,
please enable it by clicking the button as shown in the image below. See also
[Managing Github Actions Settings for a
repository](https://docs.github.com/en/repositories/managing-your-repositorys-settings-and-features/enabling-features-for-your-repository/managing-github-actions-settings-for-a-repository)
for more details.
- 2. It is possible your branch is based on the old \`master\`
branch in Apache Spark, please sync your branch to the latest master branch.
For example as below:
- \`\`\`bash
- git fetch upstream
- git rebase upstream/master
- git push origin YOUR_BRANCH --force
- \`\`\``,
- images: [
- {
- alt: 'enabling workflows button',
- image_url:
'https://raw.githubusercontent.com/apache/spark/master/.github/workflows/images/workflow-enable-button.png'
- }
- ]
- }
- })
- } else {
- const run_id = matched_run.id
-
- const actions_url = 'https://github.com/'
- + context.payload.pull_request.head.repo.full_name
- + '/actions/runs/'
- + run_id
- console.log('Actions URL: ' + actions_url)
-
- // Here we get the check run ID to provide a Check run view
instead of the
- // Actions view, see also SPARK-37879. The check run may not
have materialized
- // yet (it is created later than the workflow run, especially
when the matrix
- // is queued), so this is best-effort: if it cannot be found, we
fall back to
- // the Actions run URL rather than failing and leaving the PR
with no Build
- // check for the scheduled updater to sync.
- let retryCount = 0;
- let check_run_head;
- while (retryCount < 3) {
- const check_runs = await github.request(check_run_endpoint,
check_run_params);
- check_run_head = check_runs.data.check_runs.find(
- r => r.name === "Run / Check changes" && r.head_sha ==
head_sha);
- if (check_run_head) {
- break;
- }
- retryCount++;
- if (retryCount < 3) {
- await new Promise(resolve => setTimeout(resolve, 3000));
- }
- }
-
- let summary_url = actions_url
- if (check_run_head) {
- summary_url = 'https://github.com/'
- + context.payload.pull_request.head.repo.full_name
- + '/runs/'
- + check_run_head.id
- console.log('Check run URL: ' + summary_url)
- } else {
- console.log('Check run not found; falling back to Actions URL:
' + actions_url)
- }
-
- github.rest.checks.create({
- owner: context.repo.owner,
- repo: context.repo.repo,
- name: name,
- head_sha: head_sha,
- status: status,
- output: {
- title: 'Test results',
- summary: '[See test results](' + summary_url + ')\n\n'
- + 'If the tests fail for reasons unrelated to this pull
request, '
- + 'please rerun the workflow in your forked repository.\n'
- + 'If the failures are related to this pull request, '
- + 'please investigate them and push follow-up changes.',
- text: JSON.stringify({
- owner: context.payload.pull_request.head.repo.owner.login,
- repo: context.payload.pull_request.head.repo.name,
- run_id: run_id
- })
- },
- details_url: actions_url,
- })
- }
diff --git a/.github/workflows/update_build_status.yml
b/.github/workflows/update_build_status.yml
index 1bb2775235b3..cc477ef08f5b 100644
--- a/.github/workflows/update_build_status.yml
+++ b/.github/workflows/update_build_status.yml
@@ -17,6 +17,17 @@
# under the License.
#
+# Publishes the status of each open PR's fork build as a `Build` check on the
PR.
+# Fork builds still run in contributors' forks on push (build_main.yml); this
+# workflow only reports their results. Running on a schedule avoids
+# `pull_request_target` and needs no maintainer approval for fork PRs.
+#
+# Each pass synchronizes existing `Build` checks with their fork runs. A
missing
+# check is created as `queued`, pointing at the fork run for the PR head
commit,
+# and synchronized on a later pass. If no fork run is found, an
`action_required`
+# check with guidance is created instead. GitHub may delay or drop scheduled
+# runs, so neither the initial check nor result updates are immediate.
+
name: Update build status workflow
on:
@@ -63,11 +74,11 @@ jobs:
return false;
};
- // List all check-runs for a commit. per_page=100 (not the default
30) matches
- // notify_test_workflow.yml: a SHA can accumulate more check-runs
than one page
- // (CI matrix, external checks, duplicate Build checks from
reopened PRs), which
- // could otherwise push the target Build check off the first page
and leave the PR
- // stuck in 'queued' forever.
+ // List all check-runs for a commit. github.paginate follows every
page, so the Build
+ // check is found even when a SHA accumulates many check-runs (CI
matrix, external
+ // checks, duplicate Build checks from reopened PRs); missing it
would leave the PR
+ // stuck in 'queued' forever. per_page=100 (not the default 30)
only reduces the
+ // number of requests.
const listCheckRuns = (ref) => github.paginate(
'GET /repos/{owner}/{repo}/commits/{ref}/check-runs',
{
@@ -96,8 +107,8 @@ jobs:
return params;
};
- // A Build check this updater can actually sync: not
action_required (notify writes
- // that when it missed the fork run, and it carries no run params)
and with output
+ // A Build check this updater can actually sync: not
action_required (written when
+ // no fork run was found, and it carries no run params) and with
output
// text carrying the {owner, repo, run_id} needed to fetch the
fork run. A Build check
// with empty/malformed/fieldless output can never be synced, so
it must not count as
// present - otherwise it would suppress the backfill and leave
the PR stuck with an
@@ -107,7 +118,7 @@ jobs:
&& parseRunParams(cr) != null;
// An action_required Build check: the contributor-facing "enable
Actions / rebase"
- // status notify writes when it found no fork run. It is not
syncable, but it already
+ // status written when no fork run was found. It is not syncable,
but it already
// carries the guidance the no-run backfill branch would create,
so it is "useful".
const isActionRequiredBuildCheck = (cr) =>
cr.name == 'Build' && cr.conclusion == 'action_required';
@@ -121,22 +132,20 @@ jobs:
if (pr.mergeable_state == null ||
maybeReady.includes(pr.mergeable_state)) {
const checkRuns = await listCheckRuns(pr.head.sha)
- // Does this SHA already carry an action_required Build
check? notify (or an
- // earlier pass of this updater) writes one when no fork run
was found. It is
+ // Does this SHA already carry an action_required Build
check? An earlier pass
+ // of this updater writes one when no fork run was found. It
is
// not syncable, so it never suppresses the backfill below -
which means a PR
// that permanently lacks a fork run (Actions disabled, old
master) would
// otherwise re-poll on every 15-minute pass forever. When
one is already
// present, the backfill's re-poll is unnecessary (see
below).
const hasActionRequiredBuildCheck =
checkRuns.some(isActionRequiredBuildCheck)
- // Track whether a syncable Build check exists (see
isSyncableBuildCheck).
- // notify_test_workflow.yml creates one per push; if that
job never completed
- // (e.g. cancelled while starved of an ASF runner) the check
is missing and the
- // backfill after this loop recreates it. Sync every match
(no early break): a
- // SHA can carry more than one Build check (reopened PRs, or
a backfill that
- // raced notify). Branch protection evaluates the newest
check-run of a given
- // name, so syncing only the first would leave a newer
duplicate stuck in
- // 'queued' and block the PR.
+ // Track whether a syncable Build check exists (see
isSyncableBuildCheck); if
+ // not, the backfill after this loop creates one. Sync every
match (no early
+ // break): a SHA can carry more than one Build check
(reopened PRs, or
+ // overlapping updater runs). Branch protection evaluates
the newest check-run
+ // of a given name, so syncing only the first would leave a
newer duplicate
+ // stuck in 'queued' and block the PR.
let syncableBuildCheck = false
// Build checks whose referenced fork run is permanently
gone (404). They parse
// as syncable (their output still carries valid run params)
but can never be
@@ -203,14 +212,13 @@ jobs:
}
}
- // No syncable Build check: notify_test_workflow.yml never
created one, or only
- // a stale action_required one exists. Recreate it here,
mirroring notify. Skip
- // if the head repo was deleted.
+ // No syncable Build check: none was created yet, or only a
stale action_required
+ // one exists. Create it here. Skip if the head repo was
deleted.
if (!syncableBuildCheck && pr.head.repo) {
const forkOwner = pr.head.repo.owner.login
const forkRepo = pr.head.repo.name
- // Look up the fork's build_main.yml runs for the PR
branch, matching
- // notify_test_workflow.yml. Filter by branch (head.ref),
not just head_sha:
+ // Look up the fork's build_main.yml runs for the PR
branch.
+ // Filter by branch (head.ref), not just head_sha:
// build_main.yml skips fork pushes to master (the "Sync
fork" case), so the
// same SHA can carry a skipped/unrelated run on another
branch, and a
// head_sha-only lookup could attach the Build check to
that run instead of
@@ -270,18 +278,15 @@ jobs:
// An action_required Build check already exists and we
still found no run:
// the guidance check the no-run branch below would create
is already present,
// so there is nothing to backfill. Skip the recheck
listing and the create
- // outright. (This is only reachable via the single,
sleepless lookup above,
- // so there was no ~6s window for notify to have created a
syncable check in
- // the meantime; even if it had, skipping is safe - with
no run there is
- // nothing to promote to 'queued' this pass anyway.)
+ // outright.
if (hasActionRequiredBuildCheck && !matched_run) {
console.log(' action_required Build check present;
nothing to backfill')
continue
}
// Recheck for an existing Build check immediately before
creating one. The
- // poll above can span up to ~6 seconds, during which
notify_test_workflow.yml
- // may create the check; this recheck skips the backfill
in that common case.
+ // poll above can span up to ~6 seconds, during which an
overlapping updater run
+ // may create the check; this recheck skips the backfill
in that case.
// It narrows but cannot fully close the window (there is
no atomic
// create-if-absent, and the list endpoint is eventually
consistent) - the
// sync loop above stays tolerant of duplicates so any
that slip through still
@@ -341,8 +346,8 @@ jobs:
} else {
// No run for this head commit after re-polling (empty
result, permanent
// lookup failure, or build_main.yml missing): Actions
is disabled, the
- // branch is on an old master, or the commit never
triggered a run. Mirror
- // notify's action_required check so the PR carries a
required status
+ // branch is on an old master, or the commit never
triggered a run. Create
+ // an action_required check so the PR carries a
required status
// telling the contributor how to fix it.
console.log(' No forked run for head SHA; creating
action_required check')
await github.rest.checks.create({
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]