[
https://issues.apache.org/jira/browse/SPARK-58538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jungtaek Lim resolved SPARK-58538.
----------------------------------
Fix Version/s: 4.3.0
Resolution: Fixed
Issue resolved by pull request 57740
[https://github.com/apache/spark/pull/57740]
> Add branch-4.3 CI scheduler and release integration
> ---------------------------------------------------
>
> Key: SPARK-58538
> URL: https://issues.apache.org/jira/browse/SPARK-58538
> Project: Spark
> Issue Type: Improvement
> Components: Project Infra
> Affects Versions: 4.3.0
> Reporter: Jungtaek Lim
> Assignee: Jungtaek Lim
> Priority: Major
> Labels: pull-request-available
> Fix For: 4.3.0
>
>
> {{branch-4.3}} was cut from {{branch-4.x}} for the 4.3.0 release, but the CI
> and release plumbing that {{master}} maintains for each active release branch
> has not been added for it yet.
> The current model comes from the {{branch-4.2}} work, which happened in two
> phases: SPARK-58180 (#57319) prepared the build workflows on {{branch-4.2}}
> so they are self-contained and dispatchable, then SPARK-58242 (#57408) added
> a single {{branch42_scheduler.yml}} on {{master}} and removed the old
> per-build {{build_branch42_*.yml}} files from {{master}}. That superseded the
> earlier SPARK-56710 approach, in which the per-branch build files lived on
> {{master}}. Only scheduled workflows on the default branch fire, so the
> scheduler must live on {{master}} while everything it triggers lives on the
> branch.
> h3. Work items
> On {{master}}:
> * Add {{.github/workflows/branch43_scheduler.yml}}, modelled on
> {{branch42_scheduler.yml}}: cron entries plus a {{workflow_dispatch}}
> {{target}} choice, dispatching each build workflow with {{gh workflow run
> <workflow>.yml --repo ${{ github.repository }} --ref branch-4.3}}. Cron times
> should be staggered against the existing schedulers to spread runner load.
> * Add {{branch-4.3}} to the branch list in
> {{.github/workflows/publish_snapshot.yml}} (both the {{workflow_dispatch}}
> default and the matrix default), which is currently {{'["master",
> "branch-4.x", "branch-4.2", "branch-4.1", "branch-4.0", "branch-3.5"]'}}.
> * Add a {{branch-4.3}} badge row to {{README.md}}, following the current
> release-branch form: badges point at the self-contained workflows filtered by
> {{?branch=branch-4.3}} (including {{build_main.yml}}), as SPARK-58242
> reworked the {{branch-4.2}} row to do.
> The eight workflows the scheduler dispatches are {{build_java17}},
> {{build_java21}}, {{build_java25}}, {{build_maven}}, {{build_maven_java21}},
> {{build_non_ansi}}, {{build_python_3.11}} and {{build_python_3.14}}, matching
> the {{branch-4.2}} and {{branch-4.x}} schedulers. Note
> {{build_maven_java25.yml}} exists on the branch but is deliberately not
> dispatched by any existing scheduler, so it is left out here too.
> On {{branch-4.3}} (the SPARK-58180 equivalent):
> * Remove the dormant {{schedule:}} triggers from the nine inherited
> {{build_*.yml}} workflows, leaving {{workflow_dispatch:}}. Scheduled runs
> never fire from a non-default branch, so these crons are inert, and the
> scheduler on {{master}} is what triggers them. {{branch-4.2}} has
> {{workflow_dispatch:}} only.
> * Drop the branch token from the workflow titles. The inherited names read
> e.g. {{name: "Build / Maven (branch-4.x, Scala 2.13, Hadoop 3, JDK 17)"}},
> which both collides confusingly with the real {{branch-4.x}} runs in the
> Actions UI and check-run names, and is the wrong convention: release branches
> use generic, non-branch-tagged names, e.g. {{branch-4.2}} has {{name: "Build
> / Maven (Scala 2.13, Hadoop 3, JDK 17)"}}. The branch is implied by where the
> file lives. So the token should be removed, not changed to {{branch-4.3}}.
> * Point the {{branch}} input default of both reusable workflows at this
> branch: {{build_and_test.yml}} (inherited as {{branch-4.x}}) and
> {{maven_test.yml}} (inherited as {{master}}) both to {{branch-4.3}}, so the
> callers need no explicit {{branch:}}. On {{branch-4.2}} and {{branch-4.1}}
> both default to their own branch.
> h3. Verified as needing no change
> * {{dev/create-release/release-util.sh}} derives the latest release branch by
> excluding {{master}} and {{branch-4.x}} and taking the highest remaining.
> Checked against the real remote: it already resolves to {{branch-4.3}}.
> * {{dev/merge_spark_pr.py}} was generalized by SPARK-56710 to handle any
> {{branch-4.N}} / {{branch-M.x}}; it hardcodes no branch list and its 70
> doctests pass unchanged.
> h3. Dependency
> The {{maven_test.yml}} default on {{branch-4.3}} is the {{branch-4.3}}
> instance of SPARK-58537: five callers rely on it ({{build_maven}},
> {{build_maven_java21}}, {{build_maven_java25}}, {{build_maven_java21_arm}}
> and {{build_maven_java21_macos26}}), so until it is fixed the Maven workflows
> build {{master}} rather than the branch. It must be fixed before or with the
> new scheduler, otherwise the scheduler will faithfully dispatch Maven
> workflows that silently test the wrong branch.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
