This is an automated email from the ASF dual-hosted git repository.
HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git
The following commit(s) were added to refs/heads/master by this push:
new e357add [SPARK-60900][INFRA] Make dev/create_spark_jira.py file
spark-connect-rust JIRAs on the connect-rust version line
e357add is described below
commit e357add1fc2c53e972ff4921927461234b30da0d
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Thu Nov 5 18:35:33 2026 +0900
[SPARK-60900][INFRA] Make dev/create_spark_jira.py file spark-connect-rust
JIRAs on the connect-rust version line
### What changes were proposed in this pull request?
[SPARK-60900](https://issues.apache.org/jira/browse/SPARK-60900) makes
`dev/create_spark_jira.py` file spark-connect-rust JIRAs as AGENTS.md describes:
- Affects and Fix Version/s are both set to the newest unreleased,
unarchived `connect-rust-x.y.z` version, compared numerically, so that
`connect-rust-4.10.0` is newer than `connect-rust-4.9.0`. A new `-v/--version`
option targets another version, for example a maintenance line; it accepts only
an unreleased `connect-rust-x.y.z` version and fails otherwise.
- Issues other than subtasks are linked as "is part of" that version's
release umbrella Epic (`spark-connect-rust connect-rust-x.y.z release`) when
there is exactly one Epic with exactly that summary. Otherwise the script says
so and leaves the link to be added by hand.
- Once the issue exists, the steps after its creation only warn when they
fail, so that the script still prints the issue's key and a rerun does not file
a duplicate. The Fix Version is set by an update after the creation, since
contributors may not be allowed to set it.
- `dev/merge_connect_rust_pr.py` picks its default fix version with the
same numeric order, instead of sorting version names as strings, so that both
scripts agree once `4.10.0` and `4.9.0` coexist.
AGENTS.md now describes what the script does.
### Why are the changes needed?
`detect_affected_version` only matched version names that start with a
number, so it never picked a `connect-rust-x.y.z` version: new issues got the
newest unreleased core Spark version, no Fix Version, and no umbrella link.
That puts spark-connect-rust work in core Spark's release notes, which the
version convention exists to prevent. `dev/merge_connect_rust_pr.py` already
uses the `connect-rust-x.y.z` line for the fix version.
### Does this PR introduce _any_ user-facing change?
No. This only changes maintainer scripts.
### How was this patch tested?
- New `scripts/tests/test_spark_jira_utils.py`, run by the existing
`unittest discover -s scripts/tests` CI step, checks against a fake JIRA
client: the version order (numeric, with core Spark and released or archived
versions excluded, and a failure when there is none), that `--version` must be
an unreleased `connect-rust-x.y.z` version, both versions, the umbrella link
with and without `--version`, subtasks, that only a single Epic with the exact
summary is linked, and that a failur [...]
- `ruff check` and `ruff format --check` (0.14.0, as in CI) pass on
`scripts/`.
- Filed [SPARK-60902](https://issues.apache.org/jira/browse/SPARK-60902)
with the updated script and no `--version`: it got `connect-rust-4.3.0` as
Affects and Fix Version, and the "is part of" link to SPARK-59102, the
`connect-rust-4.3.0` release umbrella. The old script would have picked a core
Spark version and left both of the latter out.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
This pull request and its description were written by Isaac.
Closes #148 from HyukjinKwon/SPARK-60900-jira-connect-rust-versions.
Authored-by: Hyukjin Kwon <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
---
AGENTS.md | 6 +-
dev/create_spark_jira.py | 7 ++
dev/merge_connect_rust_pr.py | 21 ++--
dev/spark_jira_utils.py | 94 ++++++++++++++---
scripts/tests/test_spark_jira_utils.py | 180 +++++++++++++++++++++++++++++++++
5 files changed, 280 insertions(+), 28 deletions(-)
diff --git a/AGENTS.md b/AGENTS.md
index 0515a38..289b62c 100644
--- a/AGENTS.md
+++ b/AGENTS.md
@@ -74,8 +74,9 @@ the Rust and Python sides, no stubs, green CI.
*Affects Version/s* and the *Fix Version/s* to a `connect-rust-x.y.z` version
(e.g. `connect-rust-0.1.0`, `connect-rust-4.2.0`) — never a core Spark
version.
This keeps connect-rust work in its own release notes instead of leaking into
- core Spark's. `dev/create_spark_jira.py` and `dev/merge_connect_rust_pr.py`
- already follow this; keep it consistent when editing tickets by hand.
+ core Spark's. `dev/create_spark_jira.py` (both versions, default the newest
+ unreleased `connect-rust-x.y.z`, or `--version`) and
`dev/merge_connect_rust_pr.py`
+ (fix version) already follow this; keep it consistent when editing tickets
by hand.
- **Release umbrella JIRA: one Epic per `connect-rust-x.y.z` release.** Each
release is tracked by an umbrella Epic titled `spark-connect-rust
connect-rust-x.y.z release` (e.g. `SPARK-59035` for `connect-rust-4.2.0`,
@@ -84,6 +85,7 @@ the Rust and Python sides, no stubs, green CI.
release to it with an **`is part of` (Incorporates) issue link** — the
umbrella
`incorporates` the child, the child `is part of` the umbrella — **not** the
Epic Link field. Use only that one mechanism so the two never drift apart.
+ `dev/create_spark_jira.py` adds this link for every issue it files except
subtasks.
## 1. Environment setup
diff --git a/dev/create_spark_jira.py b/dev/create_spark_jira.py
index 907f924..76ebcd1 100755
--- a/dev/create_spark_jira.py
+++ b/dev/create_spark_jira.py
@@ -38,6 +38,12 @@ def main():
help="Issue type (e.g. Bug, Improvement)",
)
parser.add_argument("-c", "--component", help="Component for the issue")
+ parser.add_argument(
+ "-v",
+ "--version",
+ help="Unreleased connect-rust-x.y.z version for Affects and Fix
Version/s, e.g. "
+ "for a maintenance line (default: the newest unreleased one)",
+ )
parser.add_argument(
"-d",
"--description",
@@ -88,6 +94,7 @@ def main():
args.component,
parent=args.parent,
issue_type=args.type,
+ version=args.version,
description=description,
)
print(jira_id)
diff --git a/dev/merge_connect_rust_pr.py b/dev/merge_connect_rust_pr.py
index e351071..9f03327 100755
--- a/dev/merge_connect_rust_pr.py
+++ b/dev/merge_connect_rust_pr.py
@@ -35,6 +35,8 @@ from urllib.request import urlopen
from urllib.request import Request
from urllib.error import HTTPError
+from spark_jira_utils import unreleased_connect_rust_versions
+
try:
import jira.client
@@ -306,19 +308,12 @@ def resolve_jira_issue(merge_branches, comment,
default_jira_id=""):
if issue.fields.assignee is None:
choose_jira_assignee(issue)
- versions = asf_jira.project_versions("SPARK")
- # Consider only connect-rust-x.y.z, unreleased, unarchived versions.
spark-connect-rust
- # ships on its own `connect-rust-*` version line rather than the core
Spark x.y.z versions,
- # so that its JIRAs are tracked in the connect-rust release notes instead
of leaking into
- # the core Spark release notes.
- versions = [
- x
- for x in versions
- if not x.raw["released"]
- and not x.raw["archived"]
- and re.match(r"connect-rust-\d+\.\d+\.\d+", x.name)
- ]
- versions = sorted(versions, key=lambda x: x.name, reverse=True)
+ # Consider only connect-rust-x.y.z, unreleased, unarchived versions,
newest first and
+ # compared numerically, as dev/create_spark_jira.py does.
spark-connect-rust ships on its
+ # own `connect-rust-*` version line rather than the core Spark x.y.z
versions, so that its
+ # JIRAs are tracked in the connect-rust release notes instead of leaking
into the core
+ # Spark release notes.
+ versions = unreleased_connect_rust_versions(asf_jira)
default_fix_versions = []
for b in merge_branches:
diff --git a/dev/spark_jira_utils.py b/dev/spark_jira_utils.py
index 67d205d..98c5e84 100644
--- a/dev/spark_jira_utils.py
+++ b/dev/spark_jira_utils.py
@@ -53,23 +53,53 @@ def get_jira_client():
)
+# spark-connect-rust ships on its own connect-rust-x.y.z version line, so that
its JIRAs
+# appear in its release notes rather than in core Spark's.
+CONNECT_RUST_VERSION = re.compile(r"connect-rust-(\d+)\.(\d+)\.(\d+)")
+
+
+def warn(msg):
+ print(msg, file=sys.stderr)
+
+
+def unreleased_connect_rust_versions(asf_jira):
+ """Return the unreleased, unarchived connect-rust-x.y.z versions, newest
first.
+
+ Versions compare numerically, so connect-rust-4.10.0 is newer than
connect-rust-4.9.0.
+ """
+ versions = []
+ for x in asf_jira.project_versions("SPARK"):
+ match = CONNECT_RUST_VERSION.fullmatch(x.name)
+ if match and not x.raw["released"] and not x.raw["archived"]:
+ versions.append((tuple(int(n) for n in match.groups()), x))
+ return [x for _, x in sorted(versions, key=lambda pair: pair[0],
reverse=True)]
+
+
def detect_affected_version(asf_jira):
- """Return the latest unreleased x.y.z version, or exit."""
- versions = asf_jira.project_versions("SPARK")
- versions = [
- x
- for x in versions
- if not x.raw["released"] and not x.raw["archived"] and
re.match(r"\d+\.\d+\.\d+", x.name)
- ]
- versions = sorted(versions, key=lambda x: x.name, reverse=True)
+ """Return the newest unreleased connect-rust-x.y.z version, or exit."""
+ versions = unreleased_connect_rust_versions(asf_jira)
if not versions:
fail(
- "Cannot detect affected version. "
+ "Cannot detect an unreleased connect-rust-x.y.z version. "
"Please create the ticket manually at %s" % JIRA_API_BASE
)
return versions[0].name
+def find_release_umbrella(asf_jira, version):
+ """Return the key of the release umbrella Epic of `version`, or None
unless exactly one.
+
+ The summary search is a fuzzy text match, so only an Epic with exactly the
umbrella
+ summary counts.
+ """
+ summary = "spark-connect-rust %s release" % version
+ issues = asf_jira.search_issues(
+ 'project = SPARK AND issuetype = Epic AND summary ~ "\\"%s\\""' %
summary
+ )
+ keys = [i.key for i in issues if i.fields.summary == summary]
+ return keys[0] if len(keys) == 1 else None
+
+
def list_components(asf_jira):
"""Print all non-archived Spark JIRA components."""
components = asf_jira.project_components("SPARK")
@@ -81,14 +111,24 @@ def list_components(asf_jira):
def create_jira_issue(
asf_jira, title, component, parent=None, issue_type=None, version=None,
description=""
):
- """Create a JIRA issue and return the issue key (e.g. SPARK-12345)."""
- affected_version = version if version else
detect_affected_version(asf_jira)
+ """Create a JIRA issue and return the issue key (e.g. SPARK-12345).
+
+ Affects and Fix Version/s are both `version`, an unreleased
connect-rust-x.y.z version,
+ by default the newest one. Issues other than subtasks are linked as "is
part of" that
+ version's release umbrella Epic, when there is exactly one. Once the issue
exists, a
+ failing later step only warns, so that its key is still returned.
+ """
+ if version:
+ if version not in [v.name for v in
unreleased_connect_rust_versions(asf_jira)]:
+ fail("%s is not an unreleased connect-rust-x.y.z version." %
version)
+ else:
+ version = detect_affected_version(asf_jira)
issue_dict = {
"project": {"key": "SPARK"},
"summary": title,
"description": description or "",
- "versions": [{"name": affected_version}],
+ "versions": [{"name": version}],
"components": [{"name": component}],
}
@@ -100,6 +140,34 @@ def create_jira_issue(
try:
new_issue = asf_jira.create_issue(fields=issue_dict)
- return new_issue.key
except Exception as e:
fail("Failed to create JIRA issue: %s" % e)
+
+ # Setting a Fix Version can need a permission that contributors lack, and
the create
+ # screen omits the field for them, so set it separately and only warn on
failure.
+ try:
+ new_issue.update(fields={"fixVersions": [{"name": version}]})
+ except Exception as e:
+ warn(
+ "Created %s, but could not set its Fix Version to %s: %s" %
(new_issue.key, version, e)
+ )
+
+ if not parent:
+ try:
+ umbrella = find_release_umbrella(asf_jira, version)
+ if umbrella:
+ # The umbrella incorporates the new issue, which "is part of"
the umbrella.
+ asf_jira.create_issue_link(
+ "Incorporates", inwardIssue=umbrella,
outwardIssue=new_issue.key
+ )
+ else:
+ warn(
+ "No single release umbrella Epic for %s; link %s to it by
hand."
+ % (version, new_issue.key)
+ )
+ except Exception as e:
+ warn(
+ "Created %s, but could not link it to the %s release umbrella:
%s"
+ % (new_issue.key, version, e)
+ )
+ return new_issue.key
diff --git a/scripts/tests/test_spark_jira_utils.py
b/scripts/tests/test_spark_jira_utils.py
new file mode 100644
index 0000000..eb66f24
--- /dev/null
+++ b/scripts/tests/test_spark_jira_utils.py
@@ -0,0 +1,180 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Check JIRA versions and umbrella links of dev/spark_jira_utils.py against a
fake JIRA."""
+
+import contextlib
+import importlib.util
+import io
+import unittest
+from pathlib import Path
+from types import SimpleNamespace
+
+SCRIPT = Path(__file__).resolve().parents[2] / "dev" / "spark_jira_utils.py"
+SPEC = importlib.util.spec_from_file_location("spark_jira_utils", SCRIPT)
+utils = importlib.util.module_from_spec(SPEC)
+SPEC.loader.exec_module(utils)
+
+
+def version(name, released=False, archived=False):
+ return SimpleNamespace(name=name, raw={"released": released, "archived":
archived})
+
+
+class FakeIssue:
+ def __init__(self, key, summary="", fields=None, fail_update=False):
+ self.key = key
+ self.fields = SimpleNamespace(summary=summary)
+ self.created_fields = fields
+ self.updates = []
+ self.fail_update = fail_update
+
+ def update(self, fields):
+ if self.fail_update:
+ raise RuntimeError("no permission to set the Fix Version")
+ self.updates.append(fields)
+
+
+class FakeJira:
+ def __init__(self, versions, epics=(), fail_update=False,
fail_search=False, fail_link=False):
+ self.versions = versions
+ self.epics = list(epics)
+ self.created = []
+ self.links = []
+ self.fail_update, self.fail_search, self.fail_link = fail_update,
fail_search, fail_link
+
+ def project_versions(self, project):
+ assert project == "SPARK"
+ return self.versions
+
+ def search_issues(self, jql):
+ # JIRA's `summary ~` text search is fuzzy, so it can return other
Epics too.
+ assert "issuetype = Epic" in jql
+ if self.fail_search:
+ raise RuntimeError("search failed")
+ return self.epics
+
+ def create_issue(self, fields):
+ issue = FakeIssue(
+ f"SPARK-{100 + len(self.created)}",
+ fields["summary"],
+ fields,
+ fail_update=self.fail_update,
+ )
+ self.created.append(issue)
+ return issue
+
+ def create_issue_link(self, link_type, inwardIssue, outwardIssue):
+ if self.fail_link:
+ raise RuntimeError("link failed")
+ self.links.append((link_type, inwardIssue, outwardIssue))
+
+
+UMBRELLA = FakeIssue("SPARK-59102", "spark-connect-rust connect-rust-4.10.0
release")
+VERSIONS = [
+ version("5.0.0"),
+ version("connect-rust-4.9.0"),
+ version("connect-rust-4.10.0"),
+ version("connect-rust-9.0.0", released=True),
+ version("connect-rust-8.0.0", archived=True),
+]
+
+
+def create(jira, **kwargs):
+ """Create an issue and return its key with what was printed to stderr."""
+ stderr = io.StringIO()
+ with contextlib.redirect_stderr(stderr):
+ key = utils.create_jira_issue(jira, "Title", "Connect", **kwargs)
+ return key, stderr.getvalue()
+
+
+class VersionTests(unittest.TestCase):
+ def test_unreleased_connect_rust_versions_newest_first(self):
+ names = [v.name for v in
utils.unreleased_connect_rust_versions(FakeJira(VERSIONS))]
+ self.assertEqual(names, ["connect-rust-4.10.0", "connect-rust-4.9.0"])
+
+ def test_newest_unreleased_connect_rust_version_compared_numerically(self):
+ self.assertEqual(utils.detect_affected_version(FakeJira(VERSIONS)),
"connect-rust-4.10.0")
+
+ def test_core_spark_versions_are_not_candidates(self):
+ with contextlib.redirect_stdout(io.StringIO()),
self.assertRaises(SystemExit):
+ utils.detect_affected_version(FakeJira([version("5.0.0"),
version("4.1.2")]))
+
+ def
test_an_explicit_version_must_be_an_unreleased_connect_rust_version(self):
+ for name in ["4.10.0", "connect-rust-9.0.0", "connect-rust-8.0.0",
"connect-rust-1.0.0"]:
+ jira = FakeJira(VERSIONS, epics=[UMBRELLA])
+ with contextlib.redirect_stdout(io.StringIO()),
self.assertRaises(SystemExit):
+ utils.create_jira_issue(jira, "Title", "Connect", version=name)
+ self.assertEqual(jira.created, [], name)
+
+
+class CreateJiraIssueTests(unittest.TestCase):
+ def test_both_versions_and_the_release_umbrella_link(self):
+ jira = FakeJira(VERSIONS, epics=[UMBRELLA])
+ key, stderr = create(jira, issue_type="Bug")
+ (issue,) = jira.created
+ self.assertEqual(key, issue.key)
+ self.assertEqual(issue.created_fields["versions"], [{"name":
"connect-rust-4.10.0"}])
+ self.assertEqual(issue.created_fields["issuetype"], {"name": "Bug"})
+ self.assertEqual(issue.updates, [{"fixVersions": [{"name":
"connect-rust-4.10.0"}]}])
+ self.assertEqual(jira.links, [("Incorporates", "SPARK-59102", key)])
+ self.assertEqual(stderr, "")
+
+ def test_explicit_version_and_its_own_umbrella(self):
+ other = FakeIssue("SPARK-59035", "spark-connect-rust
connect-rust-4.9.0 release")
+ jira = FakeJira(VERSIONS, epics=[UMBRELLA, other])
+ key, _ = create(jira, version="connect-rust-4.9.0")
+ (issue,) = jira.created
+ self.assertEqual(issue.created_fields["versions"], [{"name":
"connect-rust-4.9.0"}])
+ self.assertEqual(issue.updates, [{"fixVersions": [{"name":
"connect-rust-4.9.0"}]}])
+ self.assertEqual(jira.links, [("Incorporates", "SPARK-59035", key)])
+
+ def test_subtasks_are_not_linked_to_the_umbrella(self):
+ jira = FakeJira(VERSIONS, epics=[UMBRELLA])
+ create(jira, parent="SPARK-1")
+ (issue,) = jira.created
+ self.assertEqual(issue.created_fields["issuetype"], {"name":
"Sub-task"})
+ self.assertEqual(issue.updates, [{"fixVersions": [{"name":
"connect-rust-4.10.0"}]}])
+ self.assertEqual(jira.links, [])
+
+ def test_only_a_single_umbrella_with_the_exact_summary_is_linked(self):
+ near = FakeIssue("SPARK-2", "spark-connect-rust connect-rust-4.10.0
release notes")
+ twin = FakeIssue("SPARK-3", UMBRELLA.fields.summary)
+ for epics in ([], [near], [UMBRELLA, twin]):
+ jira = FakeJira(VERSIONS, epics=epics)
+ key, stderr = create(jira)
+ self.assertEqual(jira.links, [], epics)
+ self.assertIn("No single release umbrella Epic for
connect-rust-4.10.0", stderr)
+ self.assertIn(key, stderr)
+ jira = FakeJira(VERSIONS, epics=[near, UMBRELLA])
+ key, _ = create(jira)
+ self.assertEqual(jira.links, [("Incorporates", "SPARK-59102", key)])
+
+ def test_steps_after_the_issue_exists_warn_and_keep_the_key(self):
+ jira = FakeJira(VERSIONS, epics=[UMBRELLA], fail_update=True)
+ key, stderr = create(jira)
+ self.assertIn("could not set its Fix Version", stderr)
+ self.assertIn(key, stderr)
+ self.assertEqual(jira.links, [("Incorporates", "SPARK-59102", key)])
+ for failure in ("fail_search", "fail_link"):
+ jira = FakeJira(VERSIONS, epics=[UMBRELLA], **{failure: True})
+ key, stderr = create(jira)
+ self.assertEqual(key, jira.created[0].key)
+ self.assertIn("could not link it to the connect-rust-4.10.0
release umbrella", stderr)
+ self.assertIn(key, stderr)
+
+
+if __name__ == "__main__":
+ unittest.main()
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]