nchammas opened a new pull request, #60098: URL: https://github.com/apache/spark/pull/60098
### What changes were proposed in this pull request? Refactor the scheduled build status updater (`update_build_status.yml`) completely: - Use batched GraphQL requests instead of per-PR REST requests. - Only update PR checks that need to be changed. Do a full sync periodically to catch build updates not associated with PR updates (e.g. a job is rerun with no new commit). - Clean up the log to only show key metrics and updates made. - Move the core logic into its own JavaScript file next to the workflow. Example of cleaned up log: ```text https://github.com/apache/spark/pull/60090 1dac863: queued -> in_progress https://github.com/apache/spark/pull/59954 48a5471: in_progress -> success 726 PRs checked, 36 skipped. Build checks: 7 updated, 2 created, 681 unchanged. ``` ### Why are the changes needed? Currently, an update pass takes about 10 minutes on a 15-minute schedule. That slows down status updates and makes changes to the workflow painful to test. Nearly all of that time goes to per-PR REST reads for PRs that haven't changed in weeks, and to rewriting checks that haven't changed. Fork runs usually finish within a few hours of a push. Of the latest fork runs on 690 open PRs, 8 were rerun more than 3 days after the push. The full sync picks those up. | | Before | After | |---|---|---| | Time per pass | ~10 min | ~45-60 s (~3-5 min for a full sync) | | PRs synced per pass | ~725 (all open) | ~170 | | API reads per pass | ~1,480 REST | 7 GraphQL | | Check updates per pass | ~685 | ~3 | | Log lines per pass | ~3,400 | ~10-15 | | Runner time per day | ~16 h | ~3 h | ### Does this PR introduce _any_ user-facing change? No, not for end-users. For contributors: `Build` checks update the same way, just sooner. A fork run rerun more than 3 days after the PR was last updated shows up within 4 hours, or on the next pass after any PR activity (e.g. a comment). The one visible change: a new commit on a PR opened from a fork's `master` gets an `action_required` check telling the contributor to open the PR from another branch, instead of a `skipped` check. ### How was this patch tested? I replayed the old and new scripts read-only against live `apache/spark` PRs, intercepting writes. The new script made no write the old one didn't. All but one of the updates it skipped were no-ops. The remaining one was a stuck rerun on a PR outside the 3-day window, which the full sync covers. A full-sync replay synced all 727 open PRs. I also replayed both scripts with existing checks hidden, so every check had to be created. 724 of 725 creates matched, and each new check had its fork run's state. The one difference was a PR pushed a few minutes earlier with no fork run yet, which the new script left for the next pass. Finally, I ran quick and full syncs locally through `actions/github-script` against live PR data, intercepting writes and simulating responses, including ignored rerun updates. They checked 169 and 729 PRs in 57 seconds and 264 seconds, and logged 7 and 8 lines. All 26 focused tests passed, covering unchanged checks, applied and ignored updates, new checks, PRs from a fork's `master`, and errors. The test and dry-run harnesses load the JavaScript module directly. Replaying the 3 open PRs from a fork's `master` with their checks hidden, each got `action_required`. TODO: Dry run in my fork with a real `GITHUB_TOKEN`. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: GitHub Copilot -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
