This is an automated email from the ASF dual-hosted git repository.

HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git


The following commit(s) were added to refs/heads/master by this push:
     new e357add  [SPARK-60900][INFRA] Make dev/create_spark_jira.py file 
spark-connect-rust JIRAs on the connect-rust version line
e357add is described below

commit e357add1fc2c53e972ff4921927461234b30da0d
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Thu Nov 5 18:35:33 2026 +0900

    [SPARK-60900][INFRA] Make dev/create_spark_jira.py file spark-connect-rust 
JIRAs on the connect-rust version line
    
    ### What changes were proposed in this pull request?
    
    [SPARK-60900](https://issues.apache.org/jira/browse/SPARK-60900) makes 
`dev/create_spark_jira.py` file spark-connect-rust JIRAs as AGENTS.md describes:
    
    - Affects and Fix Version/s are both set to the newest unreleased, 
unarchived `connect-rust-x.y.z` version, compared numerically, so that 
`connect-rust-4.10.0` is newer than `connect-rust-4.9.0`. A new `-v/--version` 
option targets another version, for example a maintenance line; it accepts only 
an unreleased `connect-rust-x.y.z` version and fails otherwise.
    - Issues other than subtasks are linked as "is part of" that version's 
release umbrella Epic (`spark-connect-rust connect-rust-x.y.z release`) when 
there is exactly one Epic with exactly that summary. Otherwise the script says 
so and leaves the link to be added by hand.
    - Once the issue exists, the steps after its creation only warn when they 
fail, so that the script still prints the issue's key and a rerun does not file 
a duplicate. The Fix Version is set by an update after the creation, since 
contributors may not be allowed to set it.
    - `dev/merge_connect_rust_pr.py` picks its default fix version with the 
same numeric order, instead of sorting version names as strings, so that both 
scripts agree once `4.10.0` and `4.9.0` coexist.
    
    AGENTS.md now describes what the script does.
    
    ### Why are the changes needed?
    
    `detect_affected_version` only matched version names that start with a 
number, so it never picked a `connect-rust-x.y.z` version: new issues got the 
newest unreleased core Spark version, no Fix Version, and no umbrella link. 
That puts spark-connect-rust work in core Spark's release notes, which the 
version convention exists to prevent. `dev/merge_connect_rust_pr.py` already 
uses the `connect-rust-x.y.z` line for the fix version.
    
    ### Does this PR introduce _any_ user-facing change?
    
    No. This only changes maintainer scripts.
    
    ### How was this patch tested?
    
    - New `scripts/tests/test_spark_jira_utils.py`, run by the existing 
`unittest discover -s scripts/tests` CI step, checks against a fake JIRA 
client: the version order (numeric, with core Spark and released or archived 
versions excluded, and a failure when there is none), that `--version` must be 
an unreleased `connect-rust-x.y.z` version, both versions, the umbrella link 
with and without `--version`, subtasks, that only a single Epic with the exact 
summary is linked, and that a failur [...]
    - `ruff check` and `ruff format --check` (0.14.0, as in CI) pass on 
`scripts/`.
    - Filed [SPARK-60902](https://issues.apache.org/jira/browse/SPARK-60902) 
with the updated script and no `--version`: it got `connect-rust-4.3.0` as 
Affects and Fix Version, and the "is part of" link to SPARK-59102, the 
`connect-rust-4.3.0` release umbrella. The old script would have picked a core 
Spark version and left both of the latter out.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Opus 5.5
    
    This pull request and its description were written by Isaac.
    
    Closes #148 from HyukjinKwon/SPARK-60900-jira-connect-rust-versions.
    
    Authored-by: Hyukjin Kwon <[email protected]>
    Signed-off-by: Hyukjin Kwon <[email protected]>
---
 AGENTS.md                              |   6 +-
 dev/create_spark_jira.py               |   7 ++
 dev/merge_connect_rust_pr.py           |  21 ++--
 dev/spark_jira_utils.py                |  94 ++++++++++++++---
 scripts/tests/test_spark_jira_utils.py | 180 +++++++++++++++++++++++++++++++++
 5 files changed, 280 insertions(+), 28 deletions(-)

diff --git a/AGENTS.md b/AGENTS.md
index 0515a38..289b62c 100644
--- a/AGENTS.md
+++ b/AGENTS.md
@@ -74,8 +74,9 @@ the Rust and Python sides, no stubs, green CI.
   *Affects Version/s* and the *Fix Version/s* to a `connect-rust-x.y.z` version
   (e.g. `connect-rust-0.1.0`, `connect-rust-4.2.0`) — never a core Spark 
version.
   This keeps connect-rust work in its own release notes instead of leaking into
-  core Spark's. `dev/create_spark_jira.py` and `dev/merge_connect_rust_pr.py`
-  already follow this; keep it consistent when editing tickets by hand.
+  core Spark's. `dev/create_spark_jira.py` (both versions, default the newest
+  unreleased `connect-rust-x.y.z`, or `--version`) and 
`dev/merge_connect_rust_pr.py`
+  (fix version) already follow this; keep it consistent when editing tickets 
by hand.
 - **Release umbrella JIRA: one Epic per `connect-rust-x.y.z` release.** Each
   release is tracked by an umbrella Epic titled `spark-connect-rust
   connect-rust-x.y.z release` (e.g. `SPARK-59035` for `connect-rust-4.2.0`,
@@ -84,6 +85,7 @@ the Rust and Python sides, no stubs, green CI.
   release to it with an **`is part of` (Incorporates) issue link** — the 
umbrella
   `incorporates` the child, the child `is part of` the umbrella — **not** the
   Epic Link field. Use only that one mechanism so the two never drift apart.
+  `dev/create_spark_jira.py` adds this link for every issue it files except 
subtasks.
 
 ## 1. Environment setup
 
diff --git a/dev/create_spark_jira.py b/dev/create_spark_jira.py
index 907f924..76ebcd1 100755
--- a/dev/create_spark_jira.py
+++ b/dev/create_spark_jira.py
@@ -38,6 +38,12 @@ def main():
         help="Issue type (e.g. Bug, Improvement)",
     )
     parser.add_argument("-c", "--component", help="Component for the issue")
+    parser.add_argument(
+        "-v",
+        "--version",
+        help="Unreleased connect-rust-x.y.z version for Affects and Fix 
Version/s, e.g. "
+        "for a maintenance line (default: the newest unreleased one)",
+    )
     parser.add_argument(
         "-d",
         "--description",
@@ -88,6 +94,7 @@ def main():
         args.component,
         parent=args.parent,
         issue_type=args.type,
+        version=args.version,
         description=description,
     )
     print(jira_id)
diff --git a/dev/merge_connect_rust_pr.py b/dev/merge_connect_rust_pr.py
index e351071..9f03327 100755
--- a/dev/merge_connect_rust_pr.py
+++ b/dev/merge_connect_rust_pr.py
@@ -35,6 +35,8 @@ from urllib.request import urlopen
 from urllib.request import Request
 from urllib.error import HTTPError
 
+from spark_jira_utils import unreleased_connect_rust_versions
+
 try:
     import jira.client
 
@@ -306,19 +308,12 @@ def resolve_jira_issue(merge_branches, comment, 
default_jira_id=""):
     if issue.fields.assignee is None:
         choose_jira_assignee(issue)
 
-    versions = asf_jira.project_versions("SPARK")
-    # Consider only connect-rust-x.y.z, unreleased, unarchived versions. 
spark-connect-rust
-    # ships on its own `connect-rust-*` version line rather than the core 
Spark x.y.z versions,
-    # so that its JIRAs are tracked in the connect-rust release notes instead 
of leaking into
-    # the core Spark release notes.
-    versions = [
-        x
-        for x in versions
-        if not x.raw["released"]
-        and not x.raw["archived"]
-        and re.match(r"connect-rust-\d+\.\d+\.\d+", x.name)
-    ]
-    versions = sorted(versions, key=lambda x: x.name, reverse=True)
+    # Consider only connect-rust-x.y.z, unreleased, unarchived versions, 
newest first and
+    # compared numerically, as dev/create_spark_jira.py does. 
spark-connect-rust ships on its
+    # own `connect-rust-*` version line rather than the core Spark x.y.z 
versions, so that its
+    # JIRAs are tracked in the connect-rust release notes instead of leaking 
into the core
+    # Spark release notes.
+    versions = unreleased_connect_rust_versions(asf_jira)
 
     default_fix_versions = []
     for b in merge_branches:
diff --git a/dev/spark_jira_utils.py b/dev/spark_jira_utils.py
index 67d205d..98c5e84 100644
--- a/dev/spark_jira_utils.py
+++ b/dev/spark_jira_utils.py
@@ -53,23 +53,53 @@ def get_jira_client():
     )
 
 
+# spark-connect-rust ships on its own connect-rust-x.y.z version line, so that 
its JIRAs
+# appear in its release notes rather than in core Spark's.
+CONNECT_RUST_VERSION = re.compile(r"connect-rust-(\d+)\.(\d+)\.(\d+)")
+
+
+def warn(msg):
+    print(msg, file=sys.stderr)
+
+
+def unreleased_connect_rust_versions(asf_jira):
+    """Return the unreleased, unarchived connect-rust-x.y.z versions, newest 
first.
+
+    Versions compare numerically, so connect-rust-4.10.0 is newer than 
connect-rust-4.9.0.
+    """
+    versions = []
+    for x in asf_jira.project_versions("SPARK"):
+        match = CONNECT_RUST_VERSION.fullmatch(x.name)
+        if match and not x.raw["released"] and not x.raw["archived"]:
+            versions.append((tuple(int(n) for n in match.groups()), x))
+    return [x for _, x in sorted(versions, key=lambda pair: pair[0], 
reverse=True)]
+
+
 def detect_affected_version(asf_jira):
-    """Return the latest unreleased x.y.z version, or exit."""
-    versions = asf_jira.project_versions("SPARK")
-    versions = [
-        x
-        for x in versions
-        if not x.raw["released"] and not x.raw["archived"] and 
re.match(r"\d+\.\d+\.\d+", x.name)
-    ]
-    versions = sorted(versions, key=lambda x: x.name, reverse=True)
+    """Return the newest unreleased connect-rust-x.y.z version, or exit."""
+    versions = unreleased_connect_rust_versions(asf_jira)
     if not versions:
         fail(
-            "Cannot detect affected version. "
+            "Cannot detect an unreleased connect-rust-x.y.z version. "
             "Please create the ticket manually at %s" % JIRA_API_BASE
         )
     return versions[0].name
 
 
+def find_release_umbrella(asf_jira, version):
+    """Return the key of the release umbrella Epic of `version`, or None 
unless exactly one.
+
+    The summary search is a fuzzy text match, so only an Epic with exactly the 
umbrella
+    summary counts.
+    """
+    summary = "spark-connect-rust %s release" % version
+    issues = asf_jira.search_issues(
+        'project = SPARK AND issuetype = Epic AND summary ~ "\\"%s\\""' % 
summary
+    )
+    keys = [i.key for i in issues if i.fields.summary == summary]
+    return keys[0] if len(keys) == 1 else None
+
+
 def list_components(asf_jira):
     """Print all non-archived Spark JIRA components."""
     components = asf_jira.project_components("SPARK")
@@ -81,14 +111,24 @@ def list_components(asf_jira):
 def create_jira_issue(
     asf_jira, title, component, parent=None, issue_type=None, version=None, 
description=""
 ):
-    """Create a JIRA issue and return the issue key (e.g. SPARK-12345)."""
-    affected_version = version if version else 
detect_affected_version(asf_jira)
+    """Create a JIRA issue and return the issue key (e.g. SPARK-12345).
+
+    Affects and Fix Version/s are both `version`, an unreleased 
connect-rust-x.y.z version,
+    by default the newest one. Issues other than subtasks are linked as "is 
part of" that
+    version's release umbrella Epic, when there is exactly one. Once the issue 
exists, a
+    failing later step only warns, so that its key is still returned.
+    """
+    if version:
+        if version not in [v.name for v in 
unreleased_connect_rust_versions(asf_jira)]:
+            fail("%s is not an unreleased connect-rust-x.y.z version." % 
version)
+    else:
+        version = detect_affected_version(asf_jira)
 
     issue_dict = {
         "project": {"key": "SPARK"},
         "summary": title,
         "description": description or "",
-        "versions": [{"name": affected_version}],
+        "versions": [{"name": version}],
         "components": [{"name": component}],
     }
 
@@ -100,6 +140,34 @@ def create_jira_issue(
 
     try:
         new_issue = asf_jira.create_issue(fields=issue_dict)
-        return new_issue.key
     except Exception as e:
         fail("Failed to create JIRA issue: %s" % e)
+
+    # Setting a Fix Version can need a permission that contributors lack, and 
the create
+    # screen omits the field for them, so set it separately and only warn on 
failure.
+    try:
+        new_issue.update(fields={"fixVersions": [{"name": version}]})
+    except Exception as e:
+        warn(
+            "Created %s, but could not set its Fix Version to %s: %s" % 
(new_issue.key, version, e)
+        )
+
+    if not parent:
+        try:
+            umbrella = find_release_umbrella(asf_jira, version)
+            if umbrella:
+                # The umbrella incorporates the new issue, which "is part of" 
the umbrella.
+                asf_jira.create_issue_link(
+                    "Incorporates", inwardIssue=umbrella, 
outwardIssue=new_issue.key
+                )
+            else:
+                warn(
+                    "No single release umbrella Epic for %s; link %s to it by 
hand."
+                    % (version, new_issue.key)
+                )
+        except Exception as e:
+            warn(
+                "Created %s, but could not link it to the %s release umbrella: 
%s"
+                % (new_issue.key, version, e)
+            )
+    return new_issue.key
diff --git a/scripts/tests/test_spark_jira_utils.py 
b/scripts/tests/test_spark_jira_utils.py
new file mode 100644
index 0000000..eb66f24
--- /dev/null
+++ b/scripts/tests/test_spark_jira_utils.py
@@ -0,0 +1,180 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Check JIRA versions and umbrella links of dev/spark_jira_utils.py against a 
fake JIRA."""
+
+import contextlib
+import importlib.util
+import io
+import unittest
+from pathlib import Path
+from types import SimpleNamespace
+
+SCRIPT = Path(__file__).resolve().parents[2] / "dev" / "spark_jira_utils.py"
+SPEC = importlib.util.spec_from_file_location("spark_jira_utils", SCRIPT)
+utils = importlib.util.module_from_spec(SPEC)
+SPEC.loader.exec_module(utils)
+
+
+def version(name, released=False, archived=False):
+    return SimpleNamespace(name=name, raw={"released": released, "archived": 
archived})
+
+
+class FakeIssue:
+    def __init__(self, key, summary="", fields=None, fail_update=False):
+        self.key = key
+        self.fields = SimpleNamespace(summary=summary)
+        self.created_fields = fields
+        self.updates = []
+        self.fail_update = fail_update
+
+    def update(self, fields):
+        if self.fail_update:
+            raise RuntimeError("no permission to set the Fix Version")
+        self.updates.append(fields)
+
+
+class FakeJira:
+    def __init__(self, versions, epics=(), fail_update=False, 
fail_search=False, fail_link=False):
+        self.versions = versions
+        self.epics = list(epics)
+        self.created = []
+        self.links = []
+        self.fail_update, self.fail_search, self.fail_link = fail_update, 
fail_search, fail_link
+
+    def project_versions(self, project):
+        assert project == "SPARK"
+        return self.versions
+
+    def search_issues(self, jql):
+        # JIRA's `summary ~` text search is fuzzy, so it can return other 
Epics too.
+        assert "issuetype = Epic" in jql
+        if self.fail_search:
+            raise RuntimeError("search failed")
+        return self.epics
+
+    def create_issue(self, fields):
+        issue = FakeIssue(
+            f"SPARK-{100 + len(self.created)}",
+            fields["summary"],
+            fields,
+            fail_update=self.fail_update,
+        )
+        self.created.append(issue)
+        return issue
+
+    def create_issue_link(self, link_type, inwardIssue, outwardIssue):
+        if self.fail_link:
+            raise RuntimeError("link failed")
+        self.links.append((link_type, inwardIssue, outwardIssue))
+
+
+UMBRELLA = FakeIssue("SPARK-59102", "spark-connect-rust connect-rust-4.10.0 
release")
+VERSIONS = [
+    version("5.0.0"),
+    version("connect-rust-4.9.0"),
+    version("connect-rust-4.10.0"),
+    version("connect-rust-9.0.0", released=True),
+    version("connect-rust-8.0.0", archived=True),
+]
+
+
+def create(jira, **kwargs):
+    """Create an issue and return its key with what was printed to stderr."""
+    stderr = io.StringIO()
+    with contextlib.redirect_stderr(stderr):
+        key = utils.create_jira_issue(jira, "Title", "Connect", **kwargs)
+    return key, stderr.getvalue()
+
+
+class VersionTests(unittest.TestCase):
+    def test_unreleased_connect_rust_versions_newest_first(self):
+        names = [v.name for v in 
utils.unreleased_connect_rust_versions(FakeJira(VERSIONS))]
+        self.assertEqual(names, ["connect-rust-4.10.0", "connect-rust-4.9.0"])
+
+    def test_newest_unreleased_connect_rust_version_compared_numerically(self):
+        self.assertEqual(utils.detect_affected_version(FakeJira(VERSIONS)), 
"connect-rust-4.10.0")
+
+    def test_core_spark_versions_are_not_candidates(self):
+        with contextlib.redirect_stdout(io.StringIO()), 
self.assertRaises(SystemExit):
+            utils.detect_affected_version(FakeJira([version("5.0.0"), 
version("4.1.2")]))
+
+    def 
test_an_explicit_version_must_be_an_unreleased_connect_rust_version(self):
+        for name in ["4.10.0", "connect-rust-9.0.0", "connect-rust-8.0.0", 
"connect-rust-1.0.0"]:
+            jira = FakeJira(VERSIONS, epics=[UMBRELLA])
+            with contextlib.redirect_stdout(io.StringIO()), 
self.assertRaises(SystemExit):
+                utils.create_jira_issue(jira, "Title", "Connect", version=name)
+            self.assertEqual(jira.created, [], name)
+
+
+class CreateJiraIssueTests(unittest.TestCase):
+    def test_both_versions_and_the_release_umbrella_link(self):
+        jira = FakeJira(VERSIONS, epics=[UMBRELLA])
+        key, stderr = create(jira, issue_type="Bug")
+        (issue,) = jira.created
+        self.assertEqual(key, issue.key)
+        self.assertEqual(issue.created_fields["versions"], [{"name": 
"connect-rust-4.10.0"}])
+        self.assertEqual(issue.created_fields["issuetype"], {"name": "Bug"})
+        self.assertEqual(issue.updates, [{"fixVersions": [{"name": 
"connect-rust-4.10.0"}]}])
+        self.assertEqual(jira.links, [("Incorporates", "SPARK-59102", key)])
+        self.assertEqual(stderr, "")
+
+    def test_explicit_version_and_its_own_umbrella(self):
+        other = FakeIssue("SPARK-59035", "spark-connect-rust 
connect-rust-4.9.0 release")
+        jira = FakeJira(VERSIONS, epics=[UMBRELLA, other])
+        key, _ = create(jira, version="connect-rust-4.9.0")
+        (issue,) = jira.created
+        self.assertEqual(issue.created_fields["versions"], [{"name": 
"connect-rust-4.9.0"}])
+        self.assertEqual(issue.updates, [{"fixVersions": [{"name": 
"connect-rust-4.9.0"}]}])
+        self.assertEqual(jira.links, [("Incorporates", "SPARK-59035", key)])
+
+    def test_subtasks_are_not_linked_to_the_umbrella(self):
+        jira = FakeJira(VERSIONS, epics=[UMBRELLA])
+        create(jira, parent="SPARK-1")
+        (issue,) = jira.created
+        self.assertEqual(issue.created_fields["issuetype"], {"name": 
"Sub-task"})
+        self.assertEqual(issue.updates, [{"fixVersions": [{"name": 
"connect-rust-4.10.0"}]}])
+        self.assertEqual(jira.links, [])
+
+    def test_only_a_single_umbrella_with_the_exact_summary_is_linked(self):
+        near = FakeIssue("SPARK-2", "spark-connect-rust connect-rust-4.10.0 
release notes")
+        twin = FakeIssue("SPARK-3", UMBRELLA.fields.summary)
+        for epics in ([], [near], [UMBRELLA, twin]):
+            jira = FakeJira(VERSIONS, epics=epics)
+            key, stderr = create(jira)
+            self.assertEqual(jira.links, [], epics)
+            self.assertIn("No single release umbrella Epic for 
connect-rust-4.10.0", stderr)
+            self.assertIn(key, stderr)
+        jira = FakeJira(VERSIONS, epics=[near, UMBRELLA])
+        key, _ = create(jira)
+        self.assertEqual(jira.links, [("Incorporates", "SPARK-59102", key)])
+
+    def test_steps_after_the_issue_exists_warn_and_keep_the_key(self):
+        jira = FakeJira(VERSIONS, epics=[UMBRELLA], fail_update=True)
+        key, stderr = create(jira)
+        self.assertIn("could not set its Fix Version", stderr)
+        self.assertIn(key, stderr)
+        self.assertEqual(jira.links, [("Incorporates", "SPARK-59102", key)])
+        for failure in ("fail_search", "fail_link"):
+            jira = FakeJira(VERSIONS, epics=[UMBRELLA], **{failure: True})
+            key, stderr = create(jira)
+            self.assertEqual(key, jira.created[0].key)
+            self.assertIn("could not link it to the connect-rust-4.10.0 
release umbrella", stderr)
+            self.assertIn(key, stderr)
+
+
+if __name__ == "__main__":
+    unittest.main()


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to