[
https://issues.apache.org/jira/browse/SPARK-61159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ruifeng Zheng resolved SPARK-61159.
-----------------------------------
Fix Version/s: 5.0.0
Resolution: Fixed
Issue resolved by pull request 60216
[https://github.com/apache/spark/pull/60216]
> Reduce and stagger scheduled workflow runs
> ------------------------------------------
>
> Key: SPARK-61159
> URL: https://issues.apache.org/jira/browse/SPARK-61159
> Project: Spark
> Issue Type: Improvement
> Components: Project Infra
> Affects Versions: 5.0.0
> Reporter: Ruifeng Zheng
> Assignee: Ruifeng Zheng
> Priority: Major
> Labels: pull-request-available
> Fix For: 5.0.0
>
>
> ASF Infrastructure has asked Spark to reduce GitHub-hosted Actions runner
> consumption and simultaneous job pressure.
> Use 3 daily, 49 three-day, and 18 weekly target workloads, excluding the
> unchanged 15-minute build-status updater. Most two-day variants move to three
> days; former four- and six-day variants and the selected Python macOS26,
> minimum-dependency Python, and SparkR Windows variants run weekly. This is a
> master-only scheduling change; default-branch schedulers dispatch the
> existing target-branch workflow definitions.
> Reserve 04 and 12 UTC for daily jobs: cache 04:00, stale cleanup 04:30, Pages
> 12:30, retaining the cache-to-Pages ordering note and 8.5-hour margin. Weekly
> builds use 03:17, 09:17, 15:17, and 21:17 across all seven days, with numeric
> weekdays. Split branch-3.5 Scala 2.13 and Python 3.9 into Sunday 03:17 and
> 09:17 starts. Three-day jobs avoid these reserved hours.
> Three-day phases are 1-28/3, 2-31/3, and 3-31/3, with 16/17/16 workloads and
> Maven counts of 3/3/4. Separate the branch-4.x Maven variants and distribute
> Maven starts across each phase's day. Phase 1's cap removes 31st-to-1st
> consecutive runs; February 28 to March 1 remains a one-day gap in non-leap
> years. Monthly phases have variable intervals, while weekly triggers retain
> seven-day gaps.
> Deduplicate all 41 maintenance-branch crons using YAML anchors and
> environment aliases in dispatch conditions. Preserve manual target options,
> branches, guards, and skip-if-unchanged behavior. Keep the workflow README
> concise, with rationale and scheduling heuristics; use its existing
> single-line paragraph style and explicitly exempt the build-status updater.
> Individual intervals can be shortened later if faster feedback is needed.
> Downstream users receive SNAPSHOT artifacts for master and six maintenance
> branches less frequently: dates 3, 6, ..., 30, with gaps of three to five
> days across month boundaries. Manual publishing remains available. Main CI
> job selection, matrix parallelism, job timeouts, and released Spark behavior
> are unaffected.
> An earlier duration-weighted estimate projected roughly 35-37% scheduled
> runner-minute savings, excluding the updater, and about 19% of total CI at
> the observed usage mix. It predates the final phase-1 cap and phase
> rebalancing, which have not been reweighted against the historical duration
> measurements. Actual usage needs monitoring after rollout. For 2026, calendar
> enumeration gives 7,881 configured target starts versus 12,080 originally and
> 8,001 in the preceding PR revision; target counts do not imply equal runner
> costs.
> Validation: Actionlint 1.7.11 for all 35 managed workflows; before/after YAML
> comparisons; evaluation of all 41 scheduled/manual dispatch conditions;
> unchanged non-schedule behavior and main CI/updater; reserved-hour checks;
> 55,210 configured starts over 2,557 dates in 2026-2032 with no coincident
> triggers and 6,558 exact weekly gap checks; whitespace checks. Visualization
> is deferred.
> PR: https://github.com/apache/spark/pull/60216
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]