[ 
https://issues.apache.org/jira/browse/SPARK-61159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ruifeng Zheng reassigned SPARK-61159:
-------------------------------------

    Assignee: Ruifeng Zheng

> Reduce and stagger scheduled workflow runs
> ------------------------------------------
>
>                 Key: SPARK-61159
>                 URL: https://issues.apache.org/jira/browse/SPARK-61159
>             Project: Spark
>          Issue Type: Improvement
>          Components: Project Infra
>    Affects Versions: 5.0.0
>            Reporter: Ruifeng Zheng
>            Assignee: Ruifeng Zheng
>            Priority: Major
>              Labels: pull-request-available
>
> ASF Infrastructure has asked Spark to reduce GitHub-hosted Actions runner 
> consumption and simultaneous job pressure.
> Use 3 daily, 49 three-day, and 18 weekly target workloads, excluding the 
> unchanged 15-minute build-status updater. Most two-day variants move to three 
> days; former four- and six-day variants and the selected Python macOS26, 
> minimum-dependency Python, and SparkR Windows variants run weekly. This is a 
> master-only scheduling change; default-branch schedulers dispatch the 
> existing target-branch workflow definitions.
> Reserve 04 and 12 UTC for daily jobs: cache 04:00, stale cleanup 04:30, Pages 
> 12:30, retaining the cache-to-Pages ordering note and 8.5-hour margin. Weekly 
> builds use 03:17, 09:17, 15:17, and 21:17 across all seven days, with numeric 
> weekdays. Split branch-3.5 Scala 2.13 and Python 3.9 into Sunday 03:17 and 
> 09:17 starts. Three-day jobs avoid these reserved hours.
> Three-day phases are 1-28/3, 2-31/3, and 3-31/3, with 16/17/16 workloads and 
> Maven counts of 3/3/4. Separate the branch-4.x Maven variants and distribute 
> Maven starts across each phase's day. Phase 1's cap removes 31st-to-1st 
> consecutive runs; February 28 to March 1 remains a one-day gap in non-leap 
> years. Monthly phases have variable intervals, while weekly triggers retain 
> seven-day gaps.
> Deduplicate all 41 maintenance-branch crons using YAML anchors and 
> environment aliases in dispatch conditions. Preserve manual target options, 
> branches, guards, and skip-if-unchanged behavior. Keep the workflow README 
> concise, with rationale and scheduling heuristics; use its existing 
> single-line paragraph style and explicitly exempt the build-status updater. 
> Individual intervals can be shortened later if faster feedback is needed.
> Downstream users receive SNAPSHOT artifacts for master and six maintenance 
> branches less frequently: dates 3, 6, ..., 30, with gaps of three to five 
> days across month boundaries. Manual publishing remains available. Main CI 
> job selection, matrix parallelism, job timeouts, and released Spark behavior 
> are unaffected.
> An earlier duration-weighted estimate projected roughly 35-37% scheduled 
> runner-minute savings, excluding the updater, and about 19% of total CI at 
> the observed usage mix. It predates the final phase-1 cap and phase 
> rebalancing, which have not been reweighted against the historical duration 
> measurements. Actual usage needs monitoring after rollout. For 2026, calendar 
> enumeration gives 7,881 configured target starts versus 12,080 originally and 
> 8,001 in the preceding PR revision; target counts do not imply equal runner 
> costs.
> Validation: Actionlint 1.7.11 for all 35 managed workflows; before/after YAML 
> comparisons; evaluation of all 41 scheduled/manual dispatch conditions; 
> unchanged non-schedule behavior and main CI/updater; reserved-hour checks; 
> 55,210 configured starts over 2,557 dates in 2026-2032 with no coincident 
> triggers and 6,558 exact weekly gap checks; whitespace checks. Visualization 
> is deferred.
> PR: https://github.com/apache/spark/pull/60216



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to