[
https://issues.apache.org/jira/browse/SPARK-61159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ruifeng Zheng updated SPARK-61159:
----------------------------------
Description:
ASF Infrastructure has asked Spark to reduce GitHub-hosted Actions runner
consumption and concurrent job pressure.
Use daily, three-day, and weekly cadences. Move 52 two-day target workloads to
three-day phases and all 15 four- or six-day target workloads to weekly. Keep
the base-image cache daily at 04:17 UTC, Pages daily at 12:30 UTC, and stale
cleanup daily at 00:30 UTC. The build-status updater keeps its separate
schedule.
Reserve UTC hours 00, 04, 08, 11, 12, and 20 for workloads outside the
three-day cadence. Weekly variants use 04:17, 08:17, and 20:17. Branch 3.5
moves to Sunday 11:00 UTC, retaining its shared dispatch for both original
variants. The 52 three-day workloads use separate phase/hour slots in the
remaining 18 hours. Minute 17 is an arbitrary off-hour choice to avoid GitHub's
busy start-of-hour period.
Weekly weekday crons keep exact seven-day intervals across month boundaries.
Three-day day-of-month phases restart monthly and can have shorter gaps at
month boundaries. Some overlap is allowed, including the daily cache and one
weekly variant at 04:17. Running workflows and delayed starts can overlap
further.
The schedule does not rank workloads by importance. Build variants generally
receive the same three-day cadence, with already less frequent variants grouped
into the weekly cadence. An individual interval can be shortened later if
faster feedback is needed.
Update all six default-branch schedulers and their 41 matching branch-dispatch
conditions, preserving manual dispatch, target branches, other events, and
skip-if-unchanged checks. Document the cadence policy, weekly timetable,
reserved hours, minute choice, and allowed overlap in
.github/workflows/README.md. The scheduling change is master-only and needs no
maintenance-branch PRs.
Estimated savings are roughly 34-36% of aggregate scheduled runner-minutes,
excluding the unchanged build-status updater, and about 19% of total CI
runner-minutes at the observed usage mix. The nominal scheduled model gives
33.76%; calendar-aware annual scenarios give 35.31-35.50%; the dashboard-based
total projection is 18.80%. Assumptions: similar durations, branch activity,
and skip-if-unchanged behavior. Measurements: scheduled runs 2026-10-06 through
2026-11-04; dashboard November 1-8. Individual jobs do not execute faster;
fewer runs produce the runtime savings. No max-parallel or main CI changes are
included. Continue monitoring actual ASF runner usage after rollout.
PR: https://github.com/apache/spark/pull/60216
Validation: Actionlint for all 35 changed workflows; before/after YAML
comparisons for all 70 targets (3 daily, 52 three-day, 15 weekly); 41 dispatch
conditions; reserved-hour checks; unchanged main CI and build-status updater;
README inventory; whitespace. Enumerated 57,087 triggers across 2,557 days
(2026-2032), including 5,100 exact seven-day gaps. At most two distinct
workflow triggers coincide, only daily cache plus weekly build at reserved
04:17.
was:
ASF Infrastructure has asked Spark to reduce GitHub-hosted Actions runner
consumption and concurrent job pressure.
Reduce and stagger scheduled workloads without making any build variant more
frequent. Move the 52 two-day variants to three days, preserve all original
four- and six-day cadences, and keep the base-image cache daily at 04:17 UTC.
Preserve daily Pages at 12:30 UTC, daily stale cleanup at 00:30 UTC, the
complete original branch-3.5 scheduler, and the build-status updater.
Reserve UTC hours 00, 04, 08, 11, 12, and 20 for workloads outside the
three-day cadence. Use 04:17, 08:17, and 20:17 for four- and six-day variants.
The 52 three-day workloads use separate slots in the remaining 18 hours. Minute
17 avoids GitHub's busy start-of-hour scheduling period; it is an arbitrary
off-hour choice.
Keep day steps at least as long as the original interval. Four-day and six-day
variants use date offsets within their reserved hours. Occasional overlap is
allowed when these cycles intersect. Day-of-month offsets restart monthly, so
the cadences are approximate calendar intervals rather than strict rolling
intervals.
The affected workloads are not ranked by importance. Cadence follows the
existing interval with a three-day minimum for rescheduled builds, while
established maintenance exceptions are preserved. An individual interval can be
shortened later if faster feedback is needed.
Update the matching branch-dispatch conditions, preserving manual dispatch,
target branches, other event triggers, and skip-if-unchanged checks. Document
the reserved hours, cadence policy, minute choice, and allowed overlap in
.github/workflows/README.md.
Estimated savings are about 30% of aggregate scheduled runner-minutes and 17%
of total CI runner-minutes at the observed usage mix, assuming similar
durations and branch activity. Individual jobs do not execute faster because of
this change.
PR: https://github.com/apache/spark/pull/60216
Validation: Actionlint, before/after YAML comparisons, non-decreasing interval
checks, reserved-hour checks, and 2026-2032 calendar enumeration. There are 52
three-day, 11 four-day, four six-day, and three daily target workloads. At most
two distinct triggers coincide, at reserved 04:17 daily and reserved 20:17 on
dates 9 and 21.
> Reduce and stagger scheduled workflow runs
> ------------------------------------------
>
> Key: SPARK-61159
> URL: https://issues.apache.org/jira/browse/SPARK-61159
> Project: Spark
> Issue Type: Improvement
> Components: Project Infra
> Affects Versions: 5.0.0
> Reporter: Ruifeng Zheng
> Priority: Major
>
> ASF Infrastructure has asked Spark to reduce GitHub-hosted Actions runner
> consumption and concurrent job pressure.
> Use daily, three-day, and weekly cadences. Move 52 two-day target workloads
> to three-day phases and all 15 four- or six-day target workloads to weekly.
> Keep the base-image cache daily at 04:17 UTC, Pages daily at 12:30 UTC, and
> stale cleanup daily at 00:30 UTC. The build-status updater keeps its separate
> schedule.
> Reserve UTC hours 00, 04, 08, 11, 12, and 20 for workloads outside the
> three-day cadence. Weekly variants use 04:17, 08:17, and 20:17. Branch 3.5
> moves to Sunday 11:00 UTC, retaining its shared dispatch for both original
> variants. The 52 three-day workloads use separate phase/hour slots in the
> remaining 18 hours. Minute 17 is an arbitrary off-hour choice to avoid
> GitHub's busy start-of-hour period.
> Weekly weekday crons keep exact seven-day intervals across month boundaries.
> Three-day day-of-month phases restart monthly and can have shorter gaps at
> month boundaries. Some overlap is allowed, including the daily cache and one
> weekly variant at 04:17. Running workflows and delayed starts can overlap
> further.
> The schedule does not rank workloads by importance. Build variants generally
> receive the same three-day cadence, with already less frequent variants
> grouped into the weekly cadence. An individual interval can be shortened
> later if faster feedback is needed.
> Update all six default-branch schedulers and their 41 matching
> branch-dispatch conditions, preserving manual dispatch, target branches,
> other events, and skip-if-unchanged checks. Document the cadence policy,
> weekly timetable, reserved hours, minute choice, and allowed overlap in
> .github/workflows/README.md. The scheduling change is master-only and needs
> no maintenance-branch PRs.
> Estimated savings are roughly 34-36% of aggregate scheduled runner-minutes,
> excluding the unchanged build-status updater, and about 19% of total CI
> runner-minutes at the observed usage mix. The nominal scheduled model gives
> 33.76%; calendar-aware annual scenarios give 35.31-35.50%; the
> dashboard-based total projection is 18.80%. Assumptions: similar durations,
> branch activity, and skip-if-unchanged behavior. Measurements: scheduled runs
> 2026-10-06 through 2026-11-04; dashboard November 1-8. Individual jobs do not
> execute faster; fewer runs produce the runtime savings. No max-parallel or
> main CI changes are included. Continue monitoring actual ASF runner usage
> after rollout.
> PR: https://github.com/apache/spark/pull/60216
> Validation: Actionlint for all 35 changed workflows; before/after YAML
> comparisons for all 70 targets (3 daily, 52 three-day, 15 weekly); 41
> dispatch conditions; reserved-hour checks; unchanged main CI and build-status
> updater; README inventory; whitespace. Enumerated 57,087 triggers across
> 2,557 days (2026-2032), including 5,100 exact seven-day gaps. At most two
> distinct workflow triggers coincide, only daily cache plus weekly build at
> reserved 04:17.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
