This is an automated email from the ASF dual-hosted git repository.

zhengruifeng pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/master by this push:
     new f6165dda6631 [SPARK-61016][INFRA] Move post-merge CI sampling into the 
merge script
f6165dda6631 is described below

commit f6165dda66318fe1b3ca237bb8440023d540f189
Author: Ruifeng Zheng <[email protected]>
AuthorDate: Sat Nov 7 19:02:10 2026 +0800

    [SPARK-61016][INFRA] Move post-merge CI sampling into the merge script
    
    ### What changes were proposed in this pull request?
    
    Move deterministic post-merge CI sampling from 
`.github/workflows/build_main.yml` into
    `dev/merge_spark_pr.py`.
    
    The merge script now:
    
    - configures sampling intervals per branch, currently one in ten commits 
for `master` and
      `branch-4.x`;
    - calculates the next destination commit count from the fetched branch 
history;
    - adds GitHub's documented `skip-checks: true` trailer to sampled-out merge 
commits;
    - emits a red warning whenever post-merge CI will be skipped; and
    - removes a copied skip trailer from cherry-picked commits before applying 
the destination
      branch's independent sampling decision.
    
    The workflow-level commit-count job and its dependency are removed. Other 
branches continue to
    run push CI normally, and fork `master` synchronization remains skipped as 
before.
    
    See 
https://docs.github.com/en/actions/how-tos/manage-workflow-runs/skip-workflow-runs.
    
    ### Why are the changes needed?
    
    Moving the sampling decision into the merge script provides four advantages:
    
    1. **Centralized control across branches.** One branch-to-interval map 
controls post-merge CI
       sampling for all configured branches, instead of maintaining separate 
workflow gates on each
       branch. Sampled-out pushes also avoid starting a workflow just to decide 
whether to build.
    2. **Clearer commit status.** The current sampling job can succeed even 
when the build is skipped,
       leaving a misleading green check on the [master commit 
page](https://github.com/apache/spark/commits/master/)
       or [branch-4.x commit 
page](https://github.com/apache/spark/commits/branch-4.x). Using GitHub's
       workflow-skip directive prevents the sampled-out run from producing a 
green or red status, so
       that run no longer appears to have tested the commit.
    
    before (skipped):
    <img width="1179" height="88" alt="image" 
src="https://github.com/user-attachments/assets/05cab7ea-f365-46ee-b0b9-da7cf4d5bd6a";
 />
    
    after (skipped):
    <img width="1182" height="83" alt="image" 
src="https://github.com/user-attachments/assets/1fa4a06f-cef1-415b-9b14-54566738c395";
 />
    
    3. **Room for future controls.** Keeping the decision at merge time allows 
additional policy or
       committer controls later. For example, a committer could enforce 
post-merge CI for a particular
       merge even when it would normally be sampled out. This PR does not add 
that override yet.
    4. **Coverage beyond the build workflow.** GitHub's skip directive 
suppresses all workflows
       triggered by the merge push, so the policy can cover post-merge CI 
beyond the build workflow.
       Dongjoon's recent move of the post-merge page job to scheduled execution 
is an example of
       another workflow whose frequency needed controlling. Future 
push-triggered workflows would
       inherit the same skip decision automatically; scheduled workflows are 
unaffected.
    
    ### Does this PR introduce _any_ user-facing change?
    
    No. This changes post-merge CI scheduling only.
    
    ### How was this patch tested?
    
    - Ran `python -m doctest dev/merge_spark_pr.py` in the `spark-dev-313` 
Conda environment.
    - Ran `ruff check dev/merge_spark_pr.py` and `git diff --check`.
    - Ran `dev/merge_spark_pr.py --dry-run 60124` and verified both sampled-out 
and selected master
      commit counts, the generated commit message, suppressed push, and cleanup.
    - Created real temporary Git commits to verify that `skip-checks: true` is 
the final trailer and
      is preceded by the two empty lines required by GitHub.
    - Tested cherry-pick messages with `-sx` attribution and sign-off lines, 
including removal of an
      inherited trailer and conditional addition of the destination branch's 
trailer.
    
    - Used the updated merge script to merge 
[#60180](https://github.com/apache/spark/pull/60180): post-merge CI was skipped 
on `master` and ran on `branch-4.x`.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Codex (GPT-5)
    
    Closes #60177 from zhengruifeng/infra-merge-ci-sampling-20261105.
    
    Authored-by: Ruifeng Zheng <[email protected]>
    Signed-off-by: Ruifeng Zheng <[email protected]>
---
 .github/workflows/build_main.yml |  79 ++-----------------------------
 dev/merge_spark_pr.py            | 100 ++++++++++++++++++++++++++++++++++++++-
 2 files changed, 102 insertions(+), 77 deletions(-)

diff --git a/.github/workflows/build_main.yml b/.github/workflows/build_main.yml
index 72fe7b9c14ae..a93adae4dcbf 100644
--- a/.github/workflows/build_main.yml
+++ b/.github/workflows/build_main.yml
@@ -25,86 +25,13 @@ on:
     - '**'
 
 jobs:
-  check-commit-count:
-    name: Check commit count
-    if: github.repository == 'apache/spark' && github.ref == 
'refs/heads/master'
-    runs-on: ubuntu-slim
-    permissions:
-      contents: read
-    outputs:
-      should-run: ${{ steps.count.outputs.should-run }}
-    steps:
-      - name: Check whether this is every tenth commit
-        id: count
-        env:
-          GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
-        run: |
-          count=$(gh api graphql \
-            -f owner="$GITHUB_REPOSITORY_OWNER" \
-            -f name="${GITHUB_REPOSITORY#*/}" \
-            -f sha="$GITHUB_SHA" \
-            -f query='
-              query($owner: String!, $name: String!, $sha: GitObjectID!) {
-                repository(owner: $owner, name: $name) {
-                  object(oid: $sha) {
-                    ... on Commit { history { totalCount } }
-                  }
-                }
-              }' \
-            --jq '.data.repository.object.history.totalCount')
-          if [[ ! "$count" =~ ^[1-9][0-9]*$ ]]; then
-            echo "::error::Expected a positive commit count, got: $count"
-            exit 1
-          fi
-          remainder=$((count % 10))
-
-          if ((remainder == 0)); then
-            should_run=true
-            icon=":white_check_mark:"
-            title="Commit count matched"
-            details="Commit count $count is divisible by 10."
-          else
-            should_run=false
-            icon=":pause_button:"
-            title="Commit count not matched"
-            details="Commit count $count is not divisible by 10."
-          fi
-
-          echo "should-run=$should_run" >> "$GITHUB_OUTPUT"
-          echo "Branch: $GITHUB_REF_NAME"
-          echo "Commit: $GITHUB_SHA"
-          echo "Commit count: $count"
-          echo "Remainder: $remainder"
-          echo "Should run: $should_run"
-          echo "Details: $details"
-          echo "::notice title=$title::$details"
-          echo "### $icon Commit count check" >> "$GITHUB_STEP_SUMMARY"
-          echo "" >> "$GITHUB_STEP_SUMMARY"
-          echo "- Branch: $GITHUB_REF_NAME" >> "$GITHUB_STEP_SUMMARY"
-          echo "- Commit: $GITHUB_SHA" >> "$GITHUB_STEP_SUMMARY"
-          echo "- Commit count: $count" >> "$GITHUB_STEP_SUMMARY"
-          echo "- Remainder: $remainder" >> "$GITHUB_STEP_SUMMARY"
-          echo "- Should run: $should_run" >> "$GITHUB_STEP_SUMMARY"
-          echo "- Details: $details" >> "$GITHUB_STEP_SUMMARY"
-
   call-build-and-test:
     permissions:
       packages: write
     name: Run
-    needs: check-commit-count
     # Skip pushes to `master` on forks: the "Sync fork" button mirrors
     # apache/spark and would otherwise re-run the full build on every sync.
-    # The sampling check is skipped outside apache/spark master. GitHub 
otherwise
-    # applies an implicit success() requirement to dependencies, so 
!cancelled()
-    # allows those builds to run when the sampling job is skipped.
-    if: >-
-      (!cancelled())
-      && (github.repository == 'apache/spark' || github.ref != 
'refs/heads/master')
-      && (
-        needs.check-commit-count.result == 'skipped'
-        || (
-          needs.check-commit-count.result == 'success'
-          && needs.check-commit-count.outputs.should-run == 'true'
-        )
-      )
+    # Post-merge sampling on apache/spark master and branch-4.x is applied by
+    # dev/merge_spark_pr.py through GitHub's workflow-skip commit trailer.
+    if: github.repository == 'apache/spark' || github.ref != 
'refs/heads/master'
     uses: ./.github/workflows/build_and_test.yml
diff --git a/dev/merge_spark_pr.py b/dev/merge_spark_pr.py
index 4945e8dabff7..9c9f79923c09 100755
--- a/dev/merge_spark_pr.py
+++ b/dev/merge_spark_pr.py
@@ -112,6 +112,14 @@ SKIP_VERSION_CHECK = os.environ.get("SKIP_VERSION_CHECK", 
"")
 # Path of this script relative to the repo root, used to fetch the canonical 
copy from master.
 MERGE_SCRIPT_REPO_PATH = "dev/merge_spark_pr.py"
 
+# Run the push-triggered post-merge workflows at each branch's configured 
commit interval.
+# Other commits get GitHub's documented workflow-skip trailer after a visible 
warning.
+# See 
https://docs.github.com/en/actions/how-tos/manage-workflow-runs/skip-workflow-runs
+POST_MERGE_CI_INTERVALS = {"master": 10, "branch-4.x": 10}
+POST_MERGE_CI_SKIP_TRAILER = "skip-checks: true"
+# Covers the trailer plus the cherry-pick attribution and committer sign-off 
added by -sx.
+POST_MERGE_CI_TRAILER_SEARCH_LENGTH = 256
+
 
 def semver_branch_rank(name):
     """
@@ -828,6 +836,79 @@ def clean_up():
             git.run("git branch -D %s" % branch)
 
 
+def should_skip_post_merge_ci(target_ref):
+    """Return whether the next commit on ``target_ref`` should skip post-merge 
CI.
+
+    The branch history count matches the check formerly performed in 
``build_main.yml``.
+    """
+    interval = POST_MERGE_CI_INTERVALS.get(target_ref)
+    if interval is None:
+        return False
+
+    commit_count = int(git.run(["git", "rev-list", "--count", 
"HEAD"]).strip()) + 1
+    if commit_count % interval == 0:
+        print(
+            "Post-merge CI will run on %s: commit count %s is divisible by %s."
+            % (target_ref, commit_count, interval)
+        )
+        return False
+
+    print_error(
+        "WARNING: Post-merge CI will be skipped on %s because commit count %s 
is not "
+        "divisible by %s. GitHub's %r trailer will be added."
+        % (target_ref, commit_count, interval, POST_MERGE_CI_SKIP_TRAILER)
+    )
+    return True
+
+
+def append_post_merge_ci_skip(message):
+    """Append the workflow-skip trailer after GitHub's required two empty 
lines.
+
+    >>> append_post_merge_ci_skip("Subject\\n")
+    'Subject\\n\\n\\nskip-checks: true\\n'
+    """
+    return "%s\n\n\n%s\n" % (message.rstrip(), POST_MERGE_CI_SKIP_TRAILER)
+
+
+def remove_inherited_post_merge_ci_skip(message):
+    """Remove an inherited skip trailer found near the end of a cherry-pick 
message.
+
+    >>> message = "Subject\\n\\nskip-checks: true\\n(cherry picked from commit 
abc)\\n"
+    >>> remove_inherited_post_merge_ci_skip(message)
+    'Subject\\n\\n(cherry picked from commit abc)\\n'
+    >>> distant = "Subject\\n\\nskip-checks: true\\n" + "x" * 256 + "\\n"
+    >>> remove_inherited_post_merge_ci_skip(distant) == distant
+    True
+    """
+    tail = message[-POST_MERGE_CI_TRAILER_SEARCH_LENGTH:]
+    marker = "\n%s\n" % POST_MERGE_CI_SKIP_TRAILER
+    marker_index = tail.rfind(marker)
+    if marker_index == -1:
+        return message
+
+    marker_index += len(message) - len(tail)
+    before = message[:marker_index].rstrip()
+    after = message[marker_index + len(marker) :].strip()
+    return "\n\n".join(part for part in (before, after) if part) + "\n"
+
+
+def amend_commit_message(message):
+    """Replace the current commit's message with the in-memory multi-line 
``message``."""
+    # The command is an argument list rather than a shell string, so -m safely 
receives the
+    # complete multi-line message without shell quoting or a temporary file.
+    git.run(
+        [
+            "git",
+            "commit",
+            "--amend",
+            "--allow-empty",
+            "--cleanup=verbatim",
+            "-m",
+            message,
+        ]
+    )
+
+
 # merge the requested PR and return the merge hash
 def merge_pr(pr_num, target_ref, title, body, pr_repo_desc, pr_author, 
co_authors):
     pr_branch_name = "%s_MERGE_PR_%s" % (BRANCH_PREFIX, pr_num)
@@ -835,6 +916,7 @@ def merge_pr(pr_num, target_ref, title, body, pr_repo_desc, 
pr_author, co_author
     git.run("git fetch %s pull/%s/head:%s" % (PR_REMOTE_NAME, pr_num, 
pr_branch_name))
     git.run("git fetch %s %s:%s" % (PUSH_REMOTE_NAME, target_ref, 
target_branch_name))
     git.run("git checkout %s" % target_branch_name)
+    skip_post_merge_ci = should_skip_post_merge_ci(target_ref)
 
     had_conflicts = False
     try:
@@ -882,7 +964,15 @@ def merge_pr(pr_num, target_ref, title, body, 
pr_repo_desc, pr_author, co_author
 
     merge_message_flags += ["-m", authors]
 
-    git.run(["git", "commit", '--author="%s"' % primary_author] + 
merge_message_flags)
+    if skip_post_merge_ci:
+        # A leading newline plus git's paragraph separator leaves the two 
empty lines which
+        # GitHub requires before the skip-checks trailer. --cleanup=verbatim 
preserves them.
+        merge_message_flags += ["-m", "\n%s" % POST_MERGE_CI_SKIP_TRAILER]
+
+    git.run(
+        ["git", "commit", "--cleanup=verbatim", '--author="%s"' % 
primary_author]
+        + merge_message_flags
+    )
 
     continue_maybe(
         "Merge complete (local ref %s). Push to %s?" % (target_branch_name, 
PUSH_REMOTE_NAME)
@@ -913,6 +1003,7 @@ def _do_cherry_pick(pr_num, merge_hash, pick_ref):
     git.run("git fetch %s %s:%s" % (PUSH_REMOTE_NAME, pick_ref, 
pick_branch_name))
     git.run("git checkout %s" % pick_branch_name)
     pick_head = git.run("git rev-parse HEAD").strip()
+    skip_post_merge_ci = should_skip_post_merge_ci(pick_ref)
 
     try:
         git.run(
@@ -946,6 +1037,13 @@ def _do_cherry_pick(pr_num, merge_hash, pick_ref):
         else:
             print("Cherry-pick already completed manually; continuing with the 
backport.")
 
+    raw_message = git.run(["git", "log", "-1", "--format=%B"])
+    updated_message = remove_inherited_post_merge_ci_skip(raw_message)
+    if skip_post_merge_ci:
+        updated_message = append_post_merge_ci_skip(updated_message)
+    if updated_message != raw_message:
+        amend_commit_message(updated_message)
+
     continue_maybe(
         "Pick complete (local ref %s). Push to %s?" % (pick_branch_name, 
PUSH_REMOTE_NAME)
     )


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to