[
https://issues.apache.org/jira/browse/SPARK-61159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ruifeng Zheng updated SPARK-61159:
----------------------------------
Description:
ASF Infrastructure has asked Spark to reduce GitHub-hosted Actions runner
consumption and concurrent job pressure.
Use daily, three-day, and weekly cadences. Move 49 two-day target workloads to
three-day phases. Run all 15 formerly four- or six-day workloads, plus Python
3.12 macOS26, minimum-dependency Python, and SparkR Windows, weekly (18
workloads). Keep the base-image cache daily at 00:17 UTC, Pages daily at 12:30
UTC, and stale cleanup daily at 00:30 UTC. The build-status updater keeps its
separate schedule.
Reserve UTC hours 00, 04, 08, 11, 12, and 20 for workloads outside the
three-day cadence. Weekly variants use 04:17, 08:17, and 20:17. Branch 3.5
moves to Sunday 11:00 UTC, retaining its shared dispatch for both original
variants. The 49 three-day workloads use separate phase/hour slots in the
remaining 18 hours. Minute 17 is an arbitrary off-hour choice to avoid GitHub's
busy start-of-hour period.
Weekly weekday crons keep exact seven-day intervals across month boundaries.
Three-day day-of-month phases restart monthly and can have shorter gaps at
month boundaries. The daily cache and stale cleanup share the 00 UTC hour, at
00:17 and 00:30 respectively. Some overlap is allowed. Running workflows and
delayed starts can overlap further.
The schedule does not assign a general importance ranking. Most build variants
use the three-day cadence; already less frequent variants and the three
explicitly selected workflows run weekly. An individual interval can be
shortened later if faster feedback is needed.
Update all six default-branch schedulers and their 41 matching branch-dispatch
conditions, preserving manual dispatch, target branches, other events, and
skip-if-unchanged checks. Document the cadence policy, weekly timetable,
reserved hours, minute choice, and allowed overlap in
.github/workflows/README.md. The scheduling change is master-only and needs no
maintenance-branch PRs.
Estimated savings are roughly 35-37% of aggregate scheduled runner-minutes,
excluding the unchanged build-status updater, and about 19% of total CI
runner-minutes at the observed usage mix. The nominal scheduled model gives
34.77%; calendar-aware annual scenarios give 36.25-36.45%; the dashboard-based
total projection is 19.30%. Assumptions: similar durations, branch activity,
and skip-if-unchanged behavior. Measurements: scheduled runs 2026-10-06 through
2026-11-04; dashboard November 1-8. Individual jobs do not execute faster;
fewer runs produce the runtime savings. No max-parallel or main CI changes are
included. Continue monitoring actual ASF runner usage after rollout.
PR: https://github.com/apache/spark/pull/60216
Validation: Actionlint for all 35 changed workflows; before/after YAML
comparisons for all 70 targets (3 daily, 49 three-day, 18 weekly); 41 dispatch
conditions; reserved-hour checks; unchanged main CI and build-status updater;
README inventory; whitespace. Enumerated 55,683 triggers across 2,557 days
(2026-2032), including 6,194 exact seven-day gaps. No distinct workflow
triggers coincide; branch 3.5 retains its shared trigger for two variants.
was:
ASF Infrastructure has asked Spark to reduce GitHub-hosted Actions runner
consumption and concurrent job pressure.
Use daily, three-day, and weekly cadences. Move 52 two-day target workloads to
three-day phases and all 15 four- or six-day target workloads to weekly. Keep
the base-image cache daily at 04:17 UTC, Pages daily at 12:30 UTC, and stale
cleanup daily at 00:30 UTC. The build-status updater keeps its separate
schedule.
Reserve UTC hours 00, 04, 08, 11, 12, and 20 for workloads outside the
three-day cadence. Weekly variants use 04:17, 08:17, and 20:17. Branch 3.5
moves to Sunday 11:00 UTC, retaining its shared dispatch for both original
variants. The 52 three-day workloads use separate phase/hour slots in the
remaining 18 hours. Minute 17 is an arbitrary off-hour choice to avoid GitHub's
busy start-of-hour period.
Weekly weekday crons keep exact seven-day intervals across month boundaries.
Three-day day-of-month phases restart monthly and can have shorter gaps at
month boundaries. Some overlap is allowed, including the daily cache and one
weekly variant at 04:17. Running workflows and delayed starts can overlap
further.
The schedule does not rank workloads by importance. Build variants generally
receive the same three-day cadence, with already less frequent variants grouped
into the weekly cadence. An individual interval can be shortened later if
faster feedback is needed.
Update all six default-branch schedulers and their 41 matching branch-dispatch
conditions, preserving manual dispatch, target branches, other events, and
skip-if-unchanged checks. Document the cadence policy, weekly timetable,
reserved hours, minute choice, and allowed overlap in
.github/workflows/README.md. The scheduling change is master-only and needs no
maintenance-branch PRs.
Estimated savings are roughly 34-36% of aggregate scheduled runner-minutes,
excluding the unchanged build-status updater, and about 19% of total CI
runner-minutes at the observed usage mix. The nominal scheduled model gives
33.76%; calendar-aware annual scenarios give 35.31-35.50%; the dashboard-based
total projection is 18.80%. Assumptions: similar durations, branch activity,
and skip-if-unchanged behavior. Measurements: scheduled runs 2026-10-06 through
2026-11-04; dashboard November 1-8. Individual jobs do not execute faster;
fewer runs produce the runtime savings. No max-parallel or main CI changes are
included. Continue monitoring actual ASF runner usage after rollout.
PR: https://github.com/apache/spark/pull/60216
Validation: Actionlint for all 35 changed workflows; before/after YAML
comparisons for all 70 targets (3 daily, 52 three-day, 15 weekly); 41 dispatch
conditions; reserved-hour checks; unchanged main CI and build-status updater;
README inventory; whitespace. Enumerated 57,087 triggers across 2,557 days
(2026-2032), including 5,100 exact seven-day gaps. At most two distinct
workflow triggers coincide, only daily cache plus weekly build at reserved
04:17.
> Reduce and stagger scheduled workflow runs
> ------------------------------------------
>
> Key: SPARK-61159
> URL: https://issues.apache.org/jira/browse/SPARK-61159
> Project: Spark
> Issue Type: Improvement
> Components: Project Infra
> Affects Versions: 5.0.0
> Reporter: Ruifeng Zheng
> Priority: Major
>
> ASF Infrastructure has asked Spark to reduce GitHub-hosted Actions runner
> consumption and concurrent job pressure.
> Use daily, three-day, and weekly cadences. Move 49 two-day target workloads
> to three-day phases. Run all 15 formerly four- or six-day workloads, plus
> Python 3.12 macOS26, minimum-dependency Python, and SparkR Windows, weekly
> (18 workloads). Keep the base-image cache daily at 00:17 UTC, Pages daily at
> 12:30 UTC, and stale cleanup daily at 00:30 UTC. The build-status updater
> keeps its separate schedule.
> Reserve UTC hours 00, 04, 08, 11, 12, and 20 for workloads outside the
> three-day cadence. Weekly variants use 04:17, 08:17, and 20:17. Branch 3.5
> moves to Sunday 11:00 UTC, retaining its shared dispatch for both original
> variants. The 49 three-day workloads use separate phase/hour slots in the
> remaining 18 hours. Minute 17 is an arbitrary off-hour choice to avoid
> GitHub's busy start-of-hour period.
> Weekly weekday crons keep exact seven-day intervals across month boundaries.
> Three-day day-of-month phases restart monthly and can have shorter gaps at
> month boundaries. The daily cache and stale cleanup share the 00 UTC hour, at
> 00:17 and 00:30 respectively. Some overlap is allowed. Running workflows and
> delayed starts can overlap further.
> The schedule does not assign a general importance ranking. Most build
> variants use the three-day cadence; already less frequent variants and the
> three explicitly selected workflows run weekly. An individual interval can be
> shortened later if faster feedback is needed.
> Update all six default-branch schedulers and their 41 matching
> branch-dispatch conditions, preserving manual dispatch, target branches,
> other events, and skip-if-unchanged checks. Document the cadence policy,
> weekly timetable, reserved hours, minute choice, and allowed overlap in
> .github/workflows/README.md. The scheduling change is master-only and needs
> no maintenance-branch PRs.
> Estimated savings are roughly 35-37% of aggregate scheduled runner-minutes,
> excluding the unchanged build-status updater, and about 19% of total CI
> runner-minutes at the observed usage mix. The nominal scheduled model gives
> 34.77%; calendar-aware annual scenarios give 36.25-36.45%; the
> dashboard-based total projection is 19.30%. Assumptions: similar durations,
> branch activity, and skip-if-unchanged behavior. Measurements: scheduled runs
> 2026-10-06 through 2026-11-04; dashboard November 1-8. Individual jobs do not
> execute faster; fewer runs produce the runtime savings. No max-parallel or
> main CI changes are included. Continue monitoring actual ASF runner usage
> after rollout.
> PR: https://github.com/apache/spark/pull/60216
> Validation: Actionlint for all 35 changed workflows; before/after YAML
> comparisons for all 70 targets (3 daily, 49 three-day, 18 weekly); 41
> dispatch conditions; reserved-hour checks; unchanged main CI and build-status
> updater; README inventory; whitespace. Enumerated 55,683 triggers across
> 2,557 days (2026-2032), including 6,194 exact seven-day gaps. No distinct
> workflow triggers coincide; branch 3.5 retains its shared trigger for two
> variants.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
