This is an automated email from the ASF dual-hosted git repository.
zhengruifeng pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new f6165dda6631 [SPARK-61016][INFRA] Move post-merge CI sampling into the
merge script
f6165dda6631 is described below
commit f6165dda66318fe1b3ca237bb8440023d540f189
Author: Ruifeng Zheng <[email protected]>
AuthorDate: Sat Nov 7 19:02:10 2026 +0800
[SPARK-61016][INFRA] Move post-merge CI sampling into the merge script
### What changes were proposed in this pull request?
Move deterministic post-merge CI sampling from
`.github/workflows/build_main.yml` into
`dev/merge_spark_pr.py`.
The merge script now:
- configures sampling intervals per branch, currently one in ten commits
for `master` and
`branch-4.x`;
- calculates the next destination commit count from the fetched branch
history;
- adds GitHub's documented `skip-checks: true` trailer to sampled-out merge
commits;
- emits a red warning whenever post-merge CI will be skipped; and
- removes a copied skip trailer from cherry-picked commits before applying
the destination
branch's independent sampling decision.
The workflow-level commit-count job and its dependency are removed. Other
branches continue to
run push CI normally, and fork `master` synchronization remains skipped as
before.
See
https://docs.github.com/en/actions/how-tos/manage-workflow-runs/skip-workflow-runs.
### Why are the changes needed?
Moving the sampling decision into the merge script provides four advantages:
1. **Centralized control across branches.** One branch-to-interval map
controls post-merge CI
sampling for all configured branches, instead of maintaining separate
workflow gates on each
branch. Sampled-out pushes also avoid starting a workflow just to decide
whether to build.
2. **Clearer commit status.** The current sampling job can succeed even
when the build is skipped,
leaving a misleading green check on the [master commit
page](https://github.com/apache/spark/commits/master/)
or [branch-4.x commit
page](https://github.com/apache/spark/commits/branch-4.x). Using GitHub's
workflow-skip directive prevents the sampled-out run from producing a
green or red status, so
that run no longer appears to have tested the commit.
before (skipped):
<img width="1179" height="88" alt="image"
src="https://github.com/user-attachments/assets/05cab7ea-f365-46ee-b0b9-da7cf4d5bd6a"
/>
after (skipped):
<img width="1182" height="83" alt="image"
src="https://github.com/user-attachments/assets/1fa4a06f-cef1-415b-9b14-54566738c395"
/>
3. **Room for future controls.** Keeping the decision at merge time allows
additional policy or
committer controls later. For example, a committer could enforce
post-merge CI for a particular
merge even when it would normally be sampled out. This PR does not add
that override yet.
4. **Coverage beyond the build workflow.** GitHub's skip directive
suppresses all workflows
triggered by the merge push, so the policy can cover post-merge CI
beyond the build workflow.
Dongjoon's recent move of the post-merge page job to scheduled execution
is an example of
another workflow whose frequency needed controlling. Future
push-triggered workflows would
inherit the same skip decision automatically; scheduled workflows are
unaffected.
### Does this PR introduce _any_ user-facing change?
No. This changes post-merge CI scheduling only.
### How was this patch tested?
- Ran `python -m doctest dev/merge_spark_pr.py` in the `spark-dev-313`
Conda environment.
- Ran `ruff check dev/merge_spark_pr.py` and `git diff --check`.
- Ran `dev/merge_spark_pr.py --dry-run 60124` and verified both sampled-out
and selected master
commit counts, the generated commit message, suppressed push, and cleanup.
- Created real temporary Git commits to verify that `skip-checks: true` is
the final trailer and
is preceded by the two empty lines required by GitHub.
- Tested cherry-pick messages with `-sx` attribution and sign-off lines,
including removal of an
inherited trailer and conditional addition of the destination branch's
trailer.
- Used the updated merge script to merge
[#60180](https://github.com/apache/spark/pull/60180): post-merge CI was skipped
on `master` and ran on `branch-4.x`.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Codex (GPT-5)
Closes #60177 from zhengruifeng/infra-merge-ci-sampling-20261105.
Authored-by: Ruifeng Zheng <[email protected]>
Signed-off-by: Ruifeng Zheng <[email protected]>
---
.github/workflows/build_main.yml | 79 ++-----------------------------
dev/merge_spark_pr.py | 100 ++++++++++++++++++++++++++++++++++++++-
2 files changed, 102 insertions(+), 77 deletions(-)
diff --git a/.github/workflows/build_main.yml b/.github/workflows/build_main.yml
index 72fe7b9c14ae..a93adae4dcbf 100644
--- a/.github/workflows/build_main.yml
+++ b/.github/workflows/build_main.yml
@@ -25,86 +25,13 @@ on:
- '**'
jobs:
- check-commit-count:
- name: Check commit count
- if: github.repository == 'apache/spark' && github.ref ==
'refs/heads/master'
- runs-on: ubuntu-slim
- permissions:
- contents: read
- outputs:
- should-run: ${{ steps.count.outputs.should-run }}
- steps:
- - name: Check whether this is every tenth commit
- id: count
- env:
- GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
- run: |
- count=$(gh api graphql \
- -f owner="$GITHUB_REPOSITORY_OWNER" \
- -f name="${GITHUB_REPOSITORY#*/}" \
- -f sha="$GITHUB_SHA" \
- -f query='
- query($owner: String!, $name: String!, $sha: GitObjectID!) {
- repository(owner: $owner, name: $name) {
- object(oid: $sha) {
- ... on Commit { history { totalCount } }
- }
- }
- }' \
- --jq '.data.repository.object.history.totalCount')
- if [[ ! "$count" =~ ^[1-9][0-9]*$ ]]; then
- echo "::error::Expected a positive commit count, got: $count"
- exit 1
- fi
- remainder=$((count % 10))
-
- if ((remainder == 0)); then
- should_run=true
- icon=":white_check_mark:"
- title="Commit count matched"
- details="Commit count $count is divisible by 10."
- else
- should_run=false
- icon=":pause_button:"
- title="Commit count not matched"
- details="Commit count $count is not divisible by 10."
- fi
-
- echo "should-run=$should_run" >> "$GITHUB_OUTPUT"
- echo "Branch: $GITHUB_REF_NAME"
- echo "Commit: $GITHUB_SHA"
- echo "Commit count: $count"
- echo "Remainder: $remainder"
- echo "Should run: $should_run"
- echo "Details: $details"
- echo "::notice title=$title::$details"
- echo "### $icon Commit count check" >> "$GITHUB_STEP_SUMMARY"
- echo "" >> "$GITHUB_STEP_SUMMARY"
- echo "- Branch: $GITHUB_REF_NAME" >> "$GITHUB_STEP_SUMMARY"
- echo "- Commit: $GITHUB_SHA" >> "$GITHUB_STEP_SUMMARY"
- echo "- Commit count: $count" >> "$GITHUB_STEP_SUMMARY"
- echo "- Remainder: $remainder" >> "$GITHUB_STEP_SUMMARY"
- echo "- Should run: $should_run" >> "$GITHUB_STEP_SUMMARY"
- echo "- Details: $details" >> "$GITHUB_STEP_SUMMARY"
-
call-build-and-test:
permissions:
packages: write
name: Run
- needs: check-commit-count
# Skip pushes to `master` on forks: the "Sync fork" button mirrors
# apache/spark and would otherwise re-run the full build on every sync.
- # The sampling check is skipped outside apache/spark master. GitHub
otherwise
- # applies an implicit success() requirement to dependencies, so
!cancelled()
- # allows those builds to run when the sampling job is skipped.
- if: >-
- (!cancelled())
- && (github.repository == 'apache/spark' || github.ref !=
'refs/heads/master')
- && (
- needs.check-commit-count.result == 'skipped'
- || (
- needs.check-commit-count.result == 'success'
- && needs.check-commit-count.outputs.should-run == 'true'
- )
- )
+ # Post-merge sampling on apache/spark master and branch-4.x is applied by
+ # dev/merge_spark_pr.py through GitHub's workflow-skip commit trailer.
+ if: github.repository == 'apache/spark' || github.ref !=
'refs/heads/master'
uses: ./.github/workflows/build_and_test.yml
diff --git a/dev/merge_spark_pr.py b/dev/merge_spark_pr.py
index 4945e8dabff7..9c9f79923c09 100755
--- a/dev/merge_spark_pr.py
+++ b/dev/merge_spark_pr.py
@@ -112,6 +112,14 @@ SKIP_VERSION_CHECK = os.environ.get("SKIP_VERSION_CHECK",
"")
# Path of this script relative to the repo root, used to fetch the canonical
copy from master.
MERGE_SCRIPT_REPO_PATH = "dev/merge_spark_pr.py"
+# Run the push-triggered post-merge workflows at each branch's configured
commit interval.
+# Other commits get GitHub's documented workflow-skip trailer after a visible
warning.
+# See
https://docs.github.com/en/actions/how-tos/manage-workflow-runs/skip-workflow-runs
+POST_MERGE_CI_INTERVALS = {"master": 10, "branch-4.x": 10}
+POST_MERGE_CI_SKIP_TRAILER = "skip-checks: true"
+# Covers the trailer plus the cherry-pick attribution and committer sign-off
added by -sx.
+POST_MERGE_CI_TRAILER_SEARCH_LENGTH = 256
+
def semver_branch_rank(name):
"""
@@ -828,6 +836,79 @@ def clean_up():
git.run("git branch -D %s" % branch)
+def should_skip_post_merge_ci(target_ref):
+ """Return whether the next commit on ``target_ref`` should skip post-merge
CI.
+
+ The branch history count matches the check formerly performed in
``build_main.yml``.
+ """
+ interval = POST_MERGE_CI_INTERVALS.get(target_ref)
+ if interval is None:
+ return False
+
+ commit_count = int(git.run(["git", "rev-list", "--count",
"HEAD"]).strip()) + 1
+ if commit_count % interval == 0:
+ print(
+ "Post-merge CI will run on %s: commit count %s is divisible by %s."
+ % (target_ref, commit_count, interval)
+ )
+ return False
+
+ print_error(
+ "WARNING: Post-merge CI will be skipped on %s because commit count %s
is not "
+ "divisible by %s. GitHub's %r trailer will be added."
+ % (target_ref, commit_count, interval, POST_MERGE_CI_SKIP_TRAILER)
+ )
+ return True
+
+
+def append_post_merge_ci_skip(message):
+ """Append the workflow-skip trailer after GitHub's required two empty
lines.
+
+ >>> append_post_merge_ci_skip("Subject\\n")
+ 'Subject\\n\\n\\nskip-checks: true\\n'
+ """
+ return "%s\n\n\n%s\n" % (message.rstrip(), POST_MERGE_CI_SKIP_TRAILER)
+
+
+def remove_inherited_post_merge_ci_skip(message):
+ """Remove an inherited skip trailer found near the end of a cherry-pick
message.
+
+ >>> message = "Subject\\n\\nskip-checks: true\\n(cherry picked from commit
abc)\\n"
+ >>> remove_inherited_post_merge_ci_skip(message)
+ 'Subject\\n\\n(cherry picked from commit abc)\\n'
+ >>> distant = "Subject\\n\\nskip-checks: true\\n" + "x" * 256 + "\\n"
+ >>> remove_inherited_post_merge_ci_skip(distant) == distant
+ True
+ """
+ tail = message[-POST_MERGE_CI_TRAILER_SEARCH_LENGTH:]
+ marker = "\n%s\n" % POST_MERGE_CI_SKIP_TRAILER
+ marker_index = tail.rfind(marker)
+ if marker_index == -1:
+ return message
+
+ marker_index += len(message) - len(tail)
+ before = message[:marker_index].rstrip()
+ after = message[marker_index + len(marker) :].strip()
+ return "\n\n".join(part for part in (before, after) if part) + "\n"
+
+
+def amend_commit_message(message):
+ """Replace the current commit's message with the in-memory multi-line
``message``."""
+ # The command is an argument list rather than a shell string, so -m safely
receives the
+ # complete multi-line message without shell quoting or a temporary file.
+ git.run(
+ [
+ "git",
+ "commit",
+ "--amend",
+ "--allow-empty",
+ "--cleanup=verbatim",
+ "-m",
+ message,
+ ]
+ )
+
+
# merge the requested PR and return the merge hash
def merge_pr(pr_num, target_ref, title, body, pr_repo_desc, pr_author,
co_authors):
pr_branch_name = "%s_MERGE_PR_%s" % (BRANCH_PREFIX, pr_num)
@@ -835,6 +916,7 @@ def merge_pr(pr_num, target_ref, title, body, pr_repo_desc,
pr_author, co_author
git.run("git fetch %s pull/%s/head:%s" % (PR_REMOTE_NAME, pr_num,
pr_branch_name))
git.run("git fetch %s %s:%s" % (PUSH_REMOTE_NAME, target_ref,
target_branch_name))
git.run("git checkout %s" % target_branch_name)
+ skip_post_merge_ci = should_skip_post_merge_ci(target_ref)
had_conflicts = False
try:
@@ -882,7 +964,15 @@ def merge_pr(pr_num, target_ref, title, body,
pr_repo_desc, pr_author, co_author
merge_message_flags += ["-m", authors]
- git.run(["git", "commit", '--author="%s"' % primary_author] +
merge_message_flags)
+ if skip_post_merge_ci:
+ # A leading newline plus git's paragraph separator leaves the two
empty lines which
+ # GitHub requires before the skip-checks trailer. --cleanup=verbatim
preserves them.
+ merge_message_flags += ["-m", "\n%s" % POST_MERGE_CI_SKIP_TRAILER]
+
+ git.run(
+ ["git", "commit", "--cleanup=verbatim", '--author="%s"' %
primary_author]
+ + merge_message_flags
+ )
continue_maybe(
"Merge complete (local ref %s). Push to %s?" % (target_branch_name,
PUSH_REMOTE_NAME)
@@ -913,6 +1003,7 @@ def _do_cherry_pick(pr_num, merge_hash, pick_ref):
git.run("git fetch %s %s:%s" % (PUSH_REMOTE_NAME, pick_ref,
pick_branch_name))
git.run("git checkout %s" % pick_branch_name)
pick_head = git.run("git rev-parse HEAD").strip()
+ skip_post_merge_ci = should_skip_post_merge_ci(pick_ref)
try:
git.run(
@@ -946,6 +1037,13 @@ def _do_cherry_pick(pr_num, merge_hash, pick_ref):
else:
print("Cherry-pick already completed manually; continuing with the
backport.")
+ raw_message = git.run(["git", "log", "-1", "--format=%B"])
+ updated_message = remove_inherited_post_merge_ci_skip(raw_message)
+ if skip_post_merge_ci:
+ updated_message = append_post_merge_ci_skip(updated_message)
+ if updated_message != raw_message:
+ amend_commit_message(updated_message)
+
continue_maybe(
"Pick complete (local ref %s). Push to %s?" % (pick_branch_name,
PUSH_REMOTE_NAME)
)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]