[
https://issues.apache.org/jira/browse/SPARK-61159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-61159:
-----------------------------------
Labels: pull-request-available (was: )
> Reduce and stagger scheduled workflow runs
> ------------------------------------------
>
> Key: SPARK-61159
> URL: https://issues.apache.org/jira/browse/SPARK-61159
> Project: Spark
> Issue Type: Improvement
> Components: Project Infra
> Affects Versions: 5.0.0
> Reporter: Ruifeng Zheng
> Priority: Major
> Labels: pull-request-available
>
> ASF Infrastructure has asked Spark to reduce GitHub-hosted Actions runner
> consumption and concurrent job pressure.
> Use daily, three-day, and weekly cadences. Move 49 two-day target workloads
> to three-day phases. Run all 15 formerly four- or six-day workloads, plus
> Python 3.12 macOS26, minimum-dependency Python, and SparkR Windows, weekly
> (18 workloads). Keep the base-image cache daily at 00:17 UTC, Pages daily at
> 01:30 UTC, and stale cleanup daily at 00:30 UTC. The build-status updater
> keeps its separate schedule.
> Reserve UTC hours 00, 01, 03, 09, 15, and 21 for workloads outside the
> three-day cadence. Weekly variants rotate through 03:17, 09:17, 15:17, and
> 21:17, retaining their weekdays. Branch 3.5 moves to Sunday 03:17 UTC,
> retaining its shared dispatch for both original variants. The 49 three-day
> workloads use separate phase/hour slots in the remaining 18 hours. Minute 17
> is an arbitrary off-hour choice to avoid GitHub's busy start-of-hour period.
> Weekly weekday crons keep exact seven-day intervals across month boundaries.
> Three-day day-of-month phases restart monthly and can have shorter gaps at
> month boundaries. The daily cache and stale cleanup share the 00 UTC hour, at
> 00:17 and 00:30 respectively. Some overlap is allowed. Running workflows and
> delayed starts can overlap further.
> The schedule does not assign a general importance ranking. Most build
> variants use the three-day cadence; already less frequent variants and the
> three explicitly selected workflows run weekly. An individual interval can be
> shortened later if faster feedback is needed.
> Update all six default-branch schedulers and their 41 matching
> branch-dispatch conditions, preserving manual dispatch, target branches,
> other events, and skip-if-unchanged checks. Document the cadence policy,
> weekly timetable, reserved hours, minute choice, and allowed overlap in
> .github/workflows/README.md. The scheduling change is master-only and needs
> no maintenance-branch PRs.
> Estimated savings are roughly 35-37% of aggregate scheduled runner-minutes,
> excluding the unchanged build-status updater, and about 19% of total CI
> runner-minutes at the observed usage mix. The nominal scheduled model gives
> 34.77%; calendar-aware annual scenarios give 36.25-36.45%; the
> dashboard-based total projection is 19.30%. Assumptions: similar durations,
> branch activity, and skip-if-unchanged behavior. Measurements: scheduled runs
> 2026-10-06 through 2026-11-04; dashboard November 1-8. Individual jobs do not
> execute faster; fewer runs produce the runtime savings. No max-parallel or
> main CI changes are included. Continue monitoring actual ASF runner usage
> after rollout.
> PR: https://github.com/apache/spark/pull/60216
> Validation: Actionlint for all 35 changed workflows; before/after YAML
> comparisons for all 70 targets (3 daily, 49 three-day, 18 weekly); 41
> dispatch conditions; reserved-hour checks; unchanged main CI and build-status
> updater; README inventory; whitespace. Enumerated 55,683 triggers across
> 2,557 days (2026-2032), including 6,194 exact seven-day gaps. No distinct
> workflow triggers coincide; branch 3.5 retains its shared trigger for two
> variants.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
