This is an automated email from the ASF dual-hosted git repository.
cloud-fan pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new 647d259c7eec [SPARK-57444][INFRA] Document versioning and branch
policy in AGENTS.md and add dev/next_version_candidates.py
647d259c7eec is described below
commit 647d259c7eec13609102af2616e143901af17d78
Author: Wenchen Fan <[email protected]>
AuthorDate: Mon Jun 15 14:05:23 2026 -0700
[SPARK-57444][INFRA] Document versioning and branch policy in AGENTS.md and
add dev/next_version_candidates.py
### What changes were proposed in this pull request?
Adds a "Versioning and Branch Policy" section to `AGENTS.md`, plus a
`dev/next_version_candidates.py` helper, so coding agents pick the correct
target version for a change (used for `since` annotations, config
`.version("...")` entries, new `MimaExcludes` sections, etc.).
The documented policy:
- Most PRs merge to **both** `master` and the latest rolling maintenance
branch `branch-<N>.x`; only breaking / binary-incompatible changes and
non-critical dependency upgrades go to `master` only.
- A change's first-release version therefore comes from the branch it first
ships in -- `branch-<N>.x` for a normally-backported change, `master` for a
master-only one -- with `-SNAPSHOT` stripped. Deriving it from `master`'s
`-SNAPSHOT` alone is the common mistake.
- Deciding whether a change is master-only is a judgement call, so the
section says to ask the user when unsure.
`dev/next_version_candidates.py` (no arguments) reports the two candidate
versions mechanically, so the caller only has to apply that judgement:
```
$ dev/next_version_candidates.py
master 5.0.0
branch-4.x 4.3.0
```
It reads from the configured `apache/spark` remote, selects the highest
`branch-<N>.x` (integer-compared, so `branch-10.x` > `branch-4.x`), strips
`-SNAPSHOT`, and reports facts only -- it does not decide which version
applies. It uses only the Python standard library and fails with an actionable
message if no `apache/spark` remote is configured (rather than fetching full
ref histories over the network into the working repo).
### Why are the changes needed?
`AGENTS.md` guides coding agents working in this repo. Without this, an
agent reading only `master` (currently `5.0.0-SNAPSHOT`) would label a
normally-backported change `since 5.0.0`, when it actually ships first in the
`branch-4.x` release (`4.3.0`). The helper removes the manual, error-prone
branch/version lookup while leaving the master-only judgement to the agent (or
the user).
### Does this PR introduce _any_ user-facing change?
No (developer tooling and documentation for coding agents only).
### How was this patch tested?
- Ran `dev/next_version_candidates.py` against the live `apache/spark`
remote -> `master 5.0.0`, `branch-4.x 4.3.0`, matching the `pom.xml` on each
branch.
- Verified the highest-`branch-<N>.x` selection end-to-end against a
throwaway remote carrying `branch-4.x`, `branch-5.x`, and `branch-10.x` (picks
`branch-10.x`), plus non-rolling decoys `branch-4.2` / `branch-5.0` (correctly
ignored).
- Verified the error path: in a repo with no `apache/spark` remote it exits
non-zero immediately with the "add a remote" message and makes no network call.
- `black` (the repo-pinned version, line length 100) reports the file
unchanged.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Opus 4.8)
This pull request and its description were written by Isaac.
Closes #56504 from cloud-fan/SPARK-agents-version-policy.
Authored-by: Wenchen Fan <[email protected]>
Signed-off-by: Wenchen Fan <[email protected]>
(cherry picked from commit d30e514d0355e745849ebaa2c5307ff81267aefe)
Signed-off-by: Wenchen Fan <[email protected]>
---
AGENTS.md | 17 +++++++
dev/next_version_candidates.py | 103 +++++++++++++++++++++++++++++++++++++++++
2 files changed, 120 insertions(+)
diff --git a/AGENTS.md b/AGENTS.md
index c37d8a130421..5bdda2578340 100644
--- a/AGENTS.md
+++ b/AGENTS.md
@@ -156,6 +156,23 @@ DO NOT force push or use `--amend` on pushed commits
unless the user explicitly
Always get user approval before external operations such as pushing commits,
creating PRs, or posting comments. Use `gh pr create` to open PRs. If `gh` is
not installed, generate the GitHub PR URL for the user and recommend installing
the GitHub CLI.
+## Versioning and Branch Policy
+
+When a change needs a version — `@since` annotations, config `.version("...")`
(`SQLConf` / `*Conf`), new `MimaExcludes` sections, etc. — use the version of
the branch it first ships in, with `-SNAPSHOT` stripped. Determine that branch:
+
+- **PR opened against a non-`master` base branch** (e.g. a maintenance line
like `branch-4.2`): use that base branch's version -- when you're checked out
on it, that's just the working tree's `pom.xml`. The helper below covers only
the common `master`-base case.
+- **PR opened against `master`:** most PRs merge to **both** `master` and the
latest `branch-<N>.x` (the branch for the next feature release, e.g.
`branch-4.x`), so use the `branch-<N>.x` version. The exception is
**master-only** changes — use `master`'s version — which are only:
+ - breaking / binary-incompatible changes that can't ship in a minor release;
+ - dependency upgrades that don't fix a critical issue worth backporting.
+
+Do **not** just read `master`'s version: a normally-backported PR ships first
in `branch-<N>.x`, whose version is lower than `master`'s. If unsure whether a
change is master-only, ask the user.
+
+`dev/next_version_candidates.py` (no arguments) prints both candidate
versions, reading from the local `apache/spark` remote (the `upstream`
configured during pre-flight). It reports the mechanical facts only -- choosing
between them per the rules above is the judgement call (the numbers below are
illustrative and advance over time):
+
+ $ dev/next_version_candidates.py
+ master 5.0.0
+ branch-4.x 4.3.0
+
## Security
Security model: [SECURITY.md](./SECURITY.md)
diff --git a/dev/next_version_candidates.py b/dev/next_version_candidates.py
new file mode 100755
index 000000000000..a63fce07e107
--- /dev/null
+++ b/dev/next_version_candidates.py
@@ -0,0 +1,103 @@
+#!/usr/bin/env python3
+
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""
+Prints the two candidate first-release versions for a change, so the caller
can pick
+one per the "Versioning and Branch Policy" section of AGENTS.md (numbers below
are
+illustrative; the actual versions advance as branches are cut):
+
+ master 5.0.0 <- use for master-only changes
+ branch-4.x 4.3.0 <- use for normally-backported changes (most PRs)
+
+Choosing between them ("is this change master-only?") is a judgement call and
is NOT
+made here -- this script only reports the mechanical facts.
+
+Takes no arguments. Reads from a local git remote pointing at apache/spark (the
+`upstream` the AGENTS.md pre-flight has you configure); errors out if none
exists,
+rather than fetching full histories over the network into your working repo.
+
+Usage: dev/next_version_candidates.py
+"""
+
+import re
+import subprocess
+import sys
+
+
+def git(*args):
+ """Runs a git command, exiting with its stderr on failure."""
+ result = subprocess.run(["git", *args], capture_output=True, text=True,
encoding="utf-8")
+ if result.returncode != 0:
+ sys.stderr.write(result.stderr)
+ sys.exit("error: command failed: git " + " ".join(args))
+ return result.stdout
+
+
+def detect_remote():
+ """Returns the name of the git remote pointing to apache/spark, or None."""
+ for line in git("remote", "-v").splitlines():
+ parts = line.split()
+ if (
+ len(parts) >= 3
+ and parts[2] == "(fetch)"
+ and re.search(r"[:/]apache/spark(\.git)?$", parts[1])
+ ):
+ return parts[0]
+ return None
+
+
+def latest_maintenance_branch(remote):
+ """Returns the highest branch-<N>.x on the remote (e.g. branch-10.x >
branch-4.x)."""
+ branches = []
+ for line in git("ls-remote", "--heads", remote,
"refs/heads/branch-*.x").splitlines():
+ name = line.rsplit("/", 1)[-1]
+ match = re.fullmatch(r"branch-(\d+)\.x", name)
+ if match:
+ branches.append((int(match.group(1)), name))
+ return max(branches)[1] if branches else None
+
+
+def pom_version(remote, ref):
+ """Fetches `ref` and returns its pom.xml project version with -SNAPSHOT
stripped."""
+ git("fetch", "--quiet", remote, ref)
+ pom = git("show", "FETCH_HEAD:pom.xml")
+ match = re.search(r"<version>([^<]+)-SNAPSHOT</version>", pom)
+ if match is None:
+ sys.exit(f"error: no -SNAPSHOT project version found in {ref}:pom.xml")
+ return match.group(1)
+
+
+def main():
+ remote = detect_remote()
+ if remote is None:
+ sys.exit(
+ "error: no git remote points to apache/spark. Add one and retry:\n"
+ " git remote add upstream https://github.com/apache/spark.git";
+ )
+
+ branch = latest_maintenance_branch(remote)
+ if branch is None:
+ sys.exit(f"error: no branch-<N>.x maintenance branch found on remote
'{remote}'")
+
+ print(f"{'master':<12} {pom_version(remote, 'master')}")
+ print(f"{branch:<12} {pom_version(remote, branch)}")
+
+
+if __name__ == "__main__":
+ main()
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
