HyukjinKwon opened a new pull request, #59872: URL: https://github.com/apache/spark/pull/59872
### What changes were proposed in this pull request? This PR lets a committer run the post-merge build of the commits they merge in their own fork, as PR builds already run in the fork of the contributor. It is opt-in per committer, and anything that does not opt in, or fails, keeps the current behavior. - `dev/merge_spark_pr.py`: with `POST_MERGE_FORK_REMOTE` set to the git remote of the committer's fork, each commit that the script pushes to apache/spark (the merge and every backport) is also force-pushed to the `post-merge-<branch>` branch of the fork. A branch whose `build_main.yml` predates this change is not pushed, and a failed push only prints an error. - Fork side (`build_and_test.yml`, `checkout-and-sync`): a push to a `post-merge-*` branch of a fork is built like a push to apache/spark is. It checks out the pushed commit as is instead of squash-merging it onto the latest apache/spark, finds the changes against the parent of the commit, runs the same jobs as apache/spark, and is not cancelled by a newer push. For the changes, the precondition job now resolves the commit to compare with and exports it as `prev_sha`, which every job uses as `GITHUB_PREV_SHA`. For every other build it is still `github.event.before`. - apache/spark side (`build_main.yml`): a new `Find post-merge build in fork` job looks up the fork of the pusher (`github.actor`) and a run of the pushed commit on its `post-merge-<branch>` branch. If it finds one, it creates a `Post-merge build` check on the commit that links the run, and the build is skipped. Otherwise, including when the job fails, the build runs here as before. - `update_build_status.yml`: a new job syncs those `Post-merge build` checks from the fork runs every 15 minutes, as the existing job does for the `Build` check of PRs. - `.github/workflows/README.md`: how a committer opts in. Pushes to a release branch keep building in apache/spark until this change is backported to it, because a push builds with the workflows of the pushed branch. ### Why are the changes needed? In the 14 days from 2026-10-14 to 2026-10-28, apache/spark used about 11,600 job-hours of GitHub Actions, about 5,800 a week. The [ASF policy](https://infra.apache.org/github-actions-policy.html) allows a project 4,200 hours a week and 3,600 hours in any five days. Post-merge builds were 57% of it: 756 runs, about 12.5 job-hours each on master and branch-4.0 to branch-4.3. PR builds already run in the forks of contributors; this lets post-merge builds run in the fork of the committer who pushed the commit. The load follows who merges: in that window one committer pushed about 40% of the commits, and a GitHub Free account runs at most 20 jobs at a time, which a committer's own PR builds share. That is why it is opt-in rather than the default. ### Does this PR introduce _any_ user-facing change? No. It changes the merge script and CI for committers who set `POST_MERGE_FORK_REMOTE`; without it, nothing changes. ### How was this patch tested? In my fork, with the two apache/spark-only jobs temporarily pointed at `HyukjinKwon/spark` (a test-only commit, not part of this PR), I pushed one commit with a one-line change to `python/pyspark/util.py` to `pm-test`, standing in for an apache/spark branch, and then to `post-merge-pm-test`, in the order the merge script pushes them: - [The `pm-test` run](https://github.com/HyukjinKwon/spark/actions/runs/40730829826) found the fork run, created [the `Post-merge build` check](https://github.com/HyukjinKwon/spark/runs/122966632797) on the commit, and skipped the build. - [The `post-merge-pm-test` run](https://github.com/HyukjinKwon/spark/actions/runs/40730868143) checked out the pushed commit as is, compared it with its parent, and selected the jobs apache/spark would: lint and the PySpark job for `pyspark-core`, without `pyspark-periodic`, docs, Java 25, buf, or the UI tests. - [The new job of `update_build_status.yml`](https://github.com/HyukjinKwon/spark/actions/runs/40731056820), dispatched by hand, moved the check from queued to in_progress while the fork run was running. `push_post_merge_build` of `dev/merge_spark_pr.py` was run against a fake git client: it does not push without `POST_MERGE_FORK_REMOTE`, or for a branch whose `build_main.yml` is missing or does not support it; it pushes for a supported branch; and a failed push only prints an error. The command that `Git.push_post_merge` runs and the output of `DryRunGit.push_post_merge` were checked too. `actionlint` with shellcheck passes on the changed workflows, `node --check` on their `github-script` blocks, and `ruff check` and `ruff format --check` on the merge script. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (Claude Opus 5.5) This pull request and its description were written by Isaac. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
