This is an automated email from the ASF dual-hosted git repository.
HyukjinKwon pushed a commit to branch branch-4.2
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.2 by this push:
new f87f2071b327 [SPARK-58136][INFRA] Automate downloads.md and
versions.json updates in the release script
f87f2071b327 is described below
commit f87f2071b327b84f13ace9210ff93fa058864d0b
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Wed Jul 15 14:04:23 2026 +0900
[SPARK-58136][INFRA] Automate downloads.md and versions.json updates in the
release script
### What changes were proposed in this pull request?
This PR adds a step to the `finalize` phase of
`dev/create-release/release-build.sh` that updates two spark-website source
files that were previously edited by hand:
- `downloads.md`: bump the Maven coordinate example (`version:`) to the
latest stable release.
- `site/static/versions.json`: add an entry for the new release, keeping
the list sorted in descending version order.
`site/downloads.html` is intentionally not edited here because it is
regenerated from `downloads.md` by the existing `bundle exec jekyll build` step.
Both updates are guarded by the same `-preview` check used by the
neighboring steps, so preview releases are not affected.
### Why are the changes needed?
The release automation already updates most of the spark-website
(`documentation.md`, `js/downloads.js`, the news post, the JIRA release notes,
the docs, and the `latest` symlink), but it did not touch `downloads.md` or
`site/static/versions.json`. As a result the release manager had to create
these edits manually, e.g. https://github.com/apache/spark-website/pull/701.
### Does this PR introduce _any_ user-facing change?
No. This only affects the release tooling.
### How was this patch tested?
Extracted the new block from the script and ran it locally against
realistic spark-website fixtures across many scenarios:
- Feature release (newest) updates both files and reproduces the manual
[spark-website#701](https://github.com/apache/spark-website/pull/701) diff.
- Preview releases (`-preview1`, `-preview2`) skip both files, matching the
sibling steps.
- Old-branch maintenance releases insert into the correct sorted slot in
`versions.json` and never downgrade the `downloads.md` coordinate.
- `downloads.md` is only bumped when the release is strictly newer than
what is shown.
- `versions.json` keeps every patch entry and is idempotent / dedup-safe on
re-runs.
- Output remains valid JSON with the existing `indent=4` formatting and
trailing newline.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Opus 4.8)
Closes #57269 from HyukjinKwon/SPARK-58136-automate-website-updates.
Authored-by: Hyukjin Kwon <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
(cherry picked from commit 74ea5d0a8f62fb376b52c7479eefd8cd23abf9ed)
Signed-off-by: Hyukjin Kwon <[email protected]>
---
dev/create-release/release-build.sh | 74 +++++++++++++++++++++++++++++++++++--
1 file changed, 70 insertions(+), 4 deletions(-)
diff --git a/dev/create-release/release-build.sh
b/dev/create-release/release-build.sh
index 8ec154ddf5da..f0cba318475a 100755
--- a/dev/create-release/release-build.sh
+++ b/dev/create-release/release-build.sh
@@ -296,7 +296,73 @@ EOF
echo "Edited js/downloads.js"
fi
- # 3. Add news post
+ # 3. Update the latest stable version in downloads.md and
site/static/versions.json.
+ # site/downloads.html is regenerated from downloads.md by the jekyll
build below,
+ # so it does not need to be edited here.
+ if [[ "$RELEASE_VERSION" =~ -preview[0-9]*$ ]]; then
+ echo "Skipping downloads.md and versions.json for preview release:
$RELEASE_VERSION"
+ else
+ python3 <<EOF
+import json
+import re
+
+release_version = "${RELEASE_VERSION}"
+
+def parse_version(v):
+ return [int(p) for p in v.strip().split(".")]
+
+def vercmp(v1, v2):
+ a = parse_version(v1)
+ b = parse_version(v2)
+ return (a > b) - (a < b)
+
+# downloads.md: bump the Maven coordinate example to the latest stable version.
+# Only update when this release is newer than the version currently shown, so a
+# maintenance release on an older branch does not downgrade the coordinate.
+with open("downloads.md") as f:
+ lines = f.readlines()
+
+with open("downloads.md", "w") as f:
+ in_maven_block = False
+ for line in lines:
+ if re.match(r"\s*artifactId:\s*spark-core", line):
+ in_maven_block = True
+ m = re.match(r"(\s*version:\s*)(\d+\.\d+\.\d+)(\s*)$", line)
+ if in_maven_block and m:
+ if vercmp(release_version, m.group(2)) > 0:
+ line = f"{m.group(1)}{release_version}{m.group(3)}"
+ in_maven_block = False
+ f.write(line)
+
+# site/static/versions.json: add (or replace) the entry for this release,
keeping
+# the list sorted in descending version order (it drives the API docs version
picker).
+path = "site/static/versions.json"
+with open(path) as f:
+ versions = json.load(f)
+
+entry = {
+ "name": release_version,
+ "version": release_version,
+ "url": f"https://spark.apache.org/docs/{release_version}/api/python/";,
+}
+
+versions = [v for v in versions if v.get("version") != release_version]
+insert_at = len(versions)
+for i, v in enumerate(versions):
+ if vercmp(release_version, v.get("version", "0.0.0")) > 0:
+ insert_at = i
+ break
+versions.insert(insert_at, entry)
+
+with open(path, "w") as f:
+ json.dump(versions, f, indent=4)
+ f.write("\n")
+EOF
+
+ echo "Edited downloads.md and site/static/versions.json"
+ fi
+
+ # 4. Add news post
RELEASE_DATE=$(TZ=America/Los_Angeles date +"%Y-%m-%d")
FILENAME="news/_posts/${RELEASE_DATE}-spark-${RELEASE_VERSION//./-}-released.md"
mkdir -p news/_posts
@@ -350,7 +416,7 @@ EOF
echo "Created $FILENAME"
- # 4. Add release notes with Python to extract JIRA version ID
+ # 5. Add release notes with Python to extract JIRA version ID
if [[ "$RELEASE_VERSION" =~ -preview[0-9]*$ ]]; then
echo "Skipping JIRA release notes for preview release: $RELEASE_VERSION"
else
@@ -424,11 +490,11 @@ EOF
echo "Created $FILENAME"
fi
- # 5. Build the website
+ # 6. Build the website
bundle install
bundle exec jekyll build
- # 6. Update latest or preview symlink
+ # 7. Update latest or preview symlink
IFS='.' read -r rel_maj rel_min rel_patch <<< "$RELEASE_VERSION"
if [[ "$RELEASE_VERSION" =~ -preview[0-9]*$ ]]; then
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
