This is an automated email from the ASF dual-hosted git repository.
dongjoon-hyun pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 6664c01e9b92 [SPARK-60468][INFRA] Only treat leading SPARK JIRA tags
as JIRA IDs in merge_spark_pr.py
6664c01e9b92 is described below
commit 6664c01e9b924f7673448ed2340099bb1795e411
Author: Kousuke Saruta <[email protected]>
AuthorDate: Fri Oct 30 07:30:36 2026 -0700
[SPARK-60468][INFRA] Only treat leading SPARK JIRA tags as JIRA IDs in
merge_spark_pr.py
### What changes were proposed in this pull request?
`dev/merge_spark_pr.py` extracts the JIRA id(s) associated with a PR from
its title. Previously it scanned the entire title with
`re.findall("SPARK-[0-9]{4,5}", title)` in three places: `resolve_jira_issues`
(resolves the ticket on merge), the revert path in `update_jira_for_pr`
(removes fix versions and offers to reopen the ticket), and
`jira_ids_to_verify` (the ids for the Epic/Umbrella blocker and JIRA-match
check in `main()`, or for the reminder printed instead when JIRA is unreac [...]
This PR adds `leading_jira_ids()` and replaces the three `re.findall` calls
with it. It returns only the ids that open the title, upper-cased and
de-duplicated:
- `Revert "..."` / `Reapply "..."` wrappers, which `main()` keeps verbatim,
are stripped first, including nested ones (`_unwrap_title()`), so those PRs
still yield the wrapped id.
- The ids are the run of `[SPARK-N]` bracket tags at the start of the
title, and scanning stops at the first other tag.
As in `Title.parse()`, an id may have any number of digits and is matched
case-insensitively. The old regex matched exactly 4 or 5 digits, so it missed
ids below `SPARK-1000` and read `SPARK-123456` as `SPARK-12345`.
For a Revert/Reapply PR, the wrapped title is an existing commit subject,
which may predate the current title rules. So two more shapes are accepted
there (for other PRs, `main()` already rejects a title that `Title.parse()`
rejects before this code runs):
- The tags after the ids are not validated, so a subject that
`Title.parse()` rejects, such as `[SPARK-48455][DOCS][MINOR] ...`, still yields
its id.
- A subject that opens with a bare id instead of a bracket, such as
`SPARK-49004`, yields only that first id.
The bracket scan is shared with `Title.parse()` through a small generator,
`_iter_bracket_tags()`.
`update_jira_for_pr` now reads the skip tags with `leading_title_tags()`,
from the same leading run of bracket tags as the ids. Previously it read them
through `Title.parse()` of the title with one `Revert "..."` level stripped,
and found none when `Title.parse()` rejected that title. Now `Reapply "..."`
and nested wrappers are stripped too, and tag aliases are normalized as
`main()` does for ordinary titles. So a `Reapply`, or a revert of a revert, of
a `[MINOR]`, `[TRIVIAL]`, or `[F [...]
- Under its registered alias `[FOLLOW-UP]`, e.g. `Revert
"[SPARK-55313][PYTHON][FOLLOW-UP] ..."`.
- In a wrapped subject that `Title.parse()` rejects, e.g.
`[SPARK-52686][FOLLOWUP][SQL][MINOR][TESTS] Fix a failing test`.
`[MINOR]` and `[TRIVIAL]` in the title skip the JIRA update only when it
has no leading id. `Title.parse()` rejects an id next to them, and the
component prompt now rejects the markers (see below), so only a subject that
predates that rule or was not merged with this script opens with an id and has
one, e.g. `[SPARK-23960][SQL][MINOR] ...` (271c891b919, 2018). Such a commit
was normally merged as the fix for its id (the skip tags date from 2026);
SPARK-23960 was resolved with fix vers [...]
When the PR title has no primary component, `prompt_for_components()` now
rejects `[MINOR]` and `[TRIVIAL]`, a SPARK id, and a token that the bracket-tag
scan cannot read back, such as `SPARK CORE`, and asks again. A PR without a
JIRA ticket puts `[MINOR]` or `[TRIVIAL]` at the start of its title, where
`Title.parse()` checks them; a PR with one cannot be marked so. Entered at the
prompt, the markers produced a title that `Title.parse()` rejects, e.g.
`[SPARK-1234][SQL][MINOR] Fix`: t [...]
The Epic/Umbrella blocker and JIRA-match check move from `main()` into
`check_linked_jira_issues()` so that they can be tested. It gets its ids from
`jira_ids_to_verify`, so both checks are still skipped when JIRA is
unreachable. Apart from the id extraction above, the only change is that the
match check now also reads `[FOLLOWUP]` (or `[FOLLOW-UP]`) inside a
`Revert`/`Reapply` wrapper. Previously such a PR could get the low title
similarity warning, although `update_jira_for_pr` skip [...]
Out of scope: `main()` still decides whether a PR is a revert from its
outermost wrapper only, so a revert of a revert takes the revert path, and a
reapply of a revert takes the resolve path. Deciding by the wrapper nesting is
left to a follow-up.
### Why are the changes needed?
A PR title may legitimately mention another ticket in its body. For example
[#59587](https://github.com/apache/spark/pull/59587):
```
[SPARK-60328][SQL][TESTS] Use a unique jar copy in the HiveDDLSuite
SPARK-34261 test
```
The old code extracted `['SPARK-60328', 'SPARK-34261']`, so merging the PR
would prompt the committer to confirm the unrelated `SPARK-34261` as a linked
ticket, and, if confirmed, would update it as well; since it is already
resolved, that means adding this PR's fix versions to it. Only the leading tag
`SPARK-60328` is the ticket this PR resolves.
### Does this PR introduce _any_ user-facing change?
No. This only affects the committer-side merge script.
### How was this patch tested?
- Doctests for `leading_jira_ids()`, `leading_title_tags()`,
`_unwrap_title()`, `_iter_bracket_tags()`, and `Component.normalize()`
(including a token that matches no component), covering body references, single
and nested `Revert`/`Reapply` wrappers, leading whitespace, duplicate ids, a
6-digit id, titles that `Title.parse()` rejects, bare ids, and titles whose id
follows another tag. The `update_jira_for_revert` doctest covers
`update_jira_for_pr` with `revert_jira_issue` and `resol [...]
- Ran `leading_jira_ids()` over all 50,656 commit subjects of master at
67b62cb4081. For every title that `Title.parse()` accepts, it returns the
parsed leading SPARK ids, de-duplicated. Of the 617 `Revert "..."` / `Reapply
"..."` subjects, only one (`Revert "[WIP] [SPARK-3996]: ..."` from 2015, whose
id follows a `[WIP]` tag) yields no id where the old regex found one, and
`main()` rejects `[WIP]` titles anyway. Only two yield fewer ids than the old
regex, and in both the dropped id [...]
- Replayed the skip-tag decision over all 50,753 commit subjects of a newer
master, 1a2450b3e5c, against master. For each of the 41,145 subjects that
`Title.parse()` accepts, `leading_title_tags()` finds the same skip tags as
`Title.parse()` with aliases normalized, which is what `main()` passes for an
ordinary title. Of the 617 `Revert "..."` / `Reapply "..."` subjects, 17 that
master would update are now skipped, all reverts of a `[FOLLOW-UP]` change, and
none that master would skip [...]
- Verified against the title of #59587: the helper returns
`['SPARK-60328']`, whereas the old regex returned `['SPARK-60328',
'SPARK-34261']`.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Kiro CLI
Closes #59708 from sarutak/SPARK-merge-pr-jira-id.
Authored-by: Kousuke Saruta <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
---
dev/merge_spark_pr.py | 564 ++++++++++++++++++++++++++++++++++++++++----------
1 file changed, 460 insertions(+), 104 deletions(-)
diff --git a/dev/merge_spark_pr.py b/dev/merge_spark_pr.py
index 9ec2333b8d75..4945e8dabff7 100755
--- a/dev/merge_spark_pr.py
+++ b/dev/merge_spark_pr.py
@@ -44,7 +44,7 @@ import re
import subprocess
import sys
import traceback
-from typing import List, Optional
+from typing import Iterator, List, Optional
from urllib.error import HTTPError
from urllib.request import Request, urlopen
@@ -1308,6 +1308,74 @@ def print_jira_issue_summary(issue):
)
+def check_linked_jira_issues(pr_num, title):
+ """Check the JIRA tickets that open ``title`` before merging, failing on a
mismatch.
+
+ Only the leading ids count (see :func:`leading_jira_ids`), so a ticket that
+ the title body mentions is neither fetched nor checked. Requires
+ ``asf_jira`` to be set unless JIRA was disabled for this run, in which case
+ both checks are skipped (see :func:`jira_ids_to_verify`).
+ """
+ jira_ids = jira_ids_to_verify(title)
+ # Epic / Umbrella tickets group related work and must not be resolved by a
single PR.
+ # Collect every offender so the committer sees the full list in one shot
rather than
+ # discovering them one-by-one across repeated merge attempts.
+ blocking_issue_types = {"Epic", "Umbrella"}
+ blockers = []
+ linked_issues = []
+ for jira_id in jira_ids:
+ try:
+ issue = asf_jira.issue(jira_id)
+ except Exception:
+ print_error("Unable to fetch summary of %s" % jira_id)
+ continue
+ print_jira_issue_summary(issue)
+ linked_issues.append((jira_id, issue))
+ issue_type = issue.fields.issuetype.name
+ if issue_type in blocking_issue_types:
+ blockers.append((jira_id, issue_type))
+ if blockers:
+ ids_str = ", ".join("%s (%s)" % (jid, t) for jid, t in blockers)
+ fail(
+ "Cannot merge PR #%s. Linked JIRA(s) %s are Umbrella or Epic "
+ "tickets and MUST not be resolved by a single PR. File "
+ "Sub-task(s) under %s and update the PR title to reference "
+ "the Sub-task(s) instead." % (pr_num, ids_str, ids_str)
+ )
+
+ # Confirm each linked JIRA actually matches this PR before merging. A
committer can
+ # reference the wrong ticket -- e.g. mistaking a GitHub PR number in the
body
+ # ("Closes #NNNNN") for a SPARK JIRA id -- which resolves an unrelated
ticket on merge.
+ # Show the PR title next to each ticket's summary, warn when a ticket is
already
+ # Resolved/Closed (a fresh merge should target an open one), and require
confirmation.
+ # For [FOLLOWUP] PRs the title describes the follow-up fix rather than the
ticket, so
+ # its low similarity to the JIRA summary is expected and the warning is
suppressed.
+ # The tag is read inside Revert/Reapply wrappers too.
+ is_followup = "FOLLOWUP" in leading_title_tags(title)
+ for jira_id, issue in linked_issues:
+ print()
+ print(
+ format_jira_verification(
+ pr_num,
+ title,
+ jira_id,
+ issue.fields.summary,
+ issue.fields.status.name,
+ issue.fields.issuetype.name,
+ resolution=(
+ issue.fields.resolution.name if issue.fields.resolution is
not None else None
+ ),
+ use_color=True,
+ is_followup=is_followup,
+ )
+ )
+ if get_input("Does %s match this PR? (y/N): " % jira_id, ["y", "n",
""]) != "y":
+ fail(
+ "Aborting: %s does not match PR #%s. Fix the PR title to
reference the "
+ "correct JIRA ticket and retry." % (jira_id, pr_num)
+ )
+
+
def jira_components_from_title_tags(tags):
"""Canonical SPARK JIRA component names implied by PR-title component tags.
@@ -1876,9 +1944,7 @@ jira_ops = None
jira_disabled = False
-def resolve_jira_issues(title, merge_branches, comment, title_components=()):
- jira_ids = re.findall("SPARK-[0-9]{4,5}", title)
-
+def resolve_jira_issues(jira_ids, merge_branches, comment,
title_components=()):
if len(jira_ids) == 0:
resolve_jira_issue(
merge_branches,
@@ -1905,14 +1971,16 @@ def update_jira_for_pr(pr_num, title, merge_branches,
title_components, is_rever
return
skip_jira_title_tags = ("MINOR", "TRIVIAL", "FOLLOWUP")
- tags = set(title_components)
- jira_title = title[len('Revert "') : -1] if is_revert else title
- try:
- parsed = Title.parse(jira_title)
- tags.update(parsed.leading)
- tags.update(parsed.components)
- except ValueError:
- pass
+ jira_ids = leading_jira_ids(title)
+ # Revert/Reapply PRs keep their title verbatim, so the tags are read
inside the wrappers,
+ # without Title.parse, which rejects some older titles such as
[SPARK-N][FOLLOWUP][MINOR].
+ tags = leading_title_tags(title)
+ if jira_ids:
+ # Title.parse rejects an id next to [MINOR] or [TRIVIAL], and
prompt_for_components()
+ # rejects both markers, so only a subject that predates the rule or
was not merged with
+ # this script has both. Such a commit was normally merged as the fix
for that id, so
+ # the id is updated.
+ tags -= _LEADING_TAGS
skipped = [tag for tag in skip_jira_title_tags if tag in tags]
if skipped:
print()
@@ -1927,7 +1995,6 @@ def update_jira_for_pr(pr_num, title, merge_branches,
title_components, is_rever
print()
continue_maybe("Would you like to update an associated JIRA?")
if is_revert:
- jira_ids = list(dict.fromkeys(re.findall("SPARK-[0-9]{4,5}",
jira_title)))
for jira_id in jira_ids or [""]:
revert_jira_issue(merge_branches, jira_id)
return
@@ -1938,7 +2005,7 @@ def update_jira_for_pr(pr_num, title, merge_branches,
title_components, is_rever
pr_num,
)
resolve_jira_issues(
- title,
+ jira_ids,
merge_branches,
jira_comment,
title_components,
@@ -1946,12 +2013,12 @@ def update_jira_for_pr(pr_num, title, merge_branches,
title_components, is_rever
def jira_ids_to_verify(title):
- """JIRA ids referenced by the PR title, for main()'s Epic/Umbrella and
match checks.
+ """Leading JIRA ids of the PR title, for check_linked_jira_issues().
Empty -- after a printed reminder to reconcile by hand -- when JIRA was
disabled for this
run, so both checks are skipped instead of failing on asf_jira being None.
"""
- jira_ids = re.findall("SPARK-[0-9]{4,5}", title)
+ jira_ids = leading_jira_ids(title)
if jira_disabled:
print_error(
f"JIRA is disabled; skipping the Epic/Umbrella check and ticket
verification. "
@@ -2001,6 +2068,16 @@ class Component:
return c
return None
+ @classmethod
+ def normalize(cls, token):
+ """Return the registered tag for ``token``, or the stripped,
upper-cased token if none.
+
+ >>> Component.normalize(" follow-up "), Component.normalize(" bogus ")
+ ('FOLLOWUP', 'BOGUS')
+ """
+ c = cls.find(token)
+ return c.tag if c is not None else token.strip().upper()
+
# Full SPARK JIRA component list (sorted alphabetically by tag), followed
# by status markers. Keep in sync with the components in JIRA — fetch the
@@ -2058,6 +2135,28 @@ _BRACKET_TAG_RE =
re.compile(r"\[\s*([A-Za-z0-9._-]+)\s*\]")
_SPARK_ID_RE = re.compile(r"^SPARK-\d+$", re.IGNORECASE)
_VERSION_TAG_RE = re.compile(r"^\d+\.(\d+|X)$")
_LEADING_TAGS = frozenset({"MINOR", "TRIVIAL"})
+# A SPARK id that opens a title without brackets, e.g. the old subject
"SPARK-49004".
+_BARE_SPARK_ID_RE = re.compile(r"SPARK-[0-9]+\b", re.IGNORECASE | re.ASCII)
+# A Revert "..." / Reapply "..." wrapper, which main() keeps verbatim in the
PR title.
+_WRAPPER_RE = re.compile(r'(?:Revert|Reapply) "(.*)"', re.DOTALL)
+
+
+def _iter_bracket_tags(raw: str) -> Iterator[tuple[str, int]]:
+ """Yield ``(TAG, end)`` for each tag in the leading run of bracket tags of
``raw``.
+
+ ``TAG`` is upper-cased, and ``end`` is the position just past the tag and
+ the spaces that follow it. Scanning stops at the first position where no
+ complete bracket tag starts.
+
+ >>> list(_iter_bracket_tags("[SPARK-1] [sql]Fix [X]"))
+ [('SPARK-1', 10), ('SQL', 15)]
+ """
+ pos = 0
+ while m := _BRACKET_TAG_RE.match(raw, pos):
+ pos = m.end()
+ while pos < len(raw) and raw[pos] == " ":
+ pos += 1
+ yield m.group(1).upper(), pos
class Title:
@@ -2156,12 +2255,8 @@ class Title:
raise ValueError("title must start with [SPARK-NNNNN], [MINOR], or
[TRIVIAL]: %r" % raw)
past_leading = False
- pos = 0
- while pos < len(raw):
- m = _BRACKET_TAG_RE.match(raw, pos)
- if not m:
- break
- tag = m.group(1).upper()
+ end = 0
+ for tag, end in _iter_bracket_tags(raw):
if _SPARK_ID_RE.match(tag):
if past_leading:
raise ValueError(
@@ -2173,11 +2268,8 @@ class Title:
else:
components.append(tag)
past_leading = True
- pos = m.end()
- while pos < len(raw) and raw[pos] == " ":
- pos += 1
- text = raw[pos:].lstrip()
+ text = raw[end:].lstrip()
markers = [t for t in leading if t in _LEADING_TAGS]
if len(markers) > 1 or (markers and len(leading) > len(markers)):
raise ValueError("[SPARK-NNNNN], [MINOR], and [TRIVIAL] cannot
coexist")
@@ -2191,14 +2283,115 @@ class Title:
return parts + (" " if parts else "") + self.text
+def _unwrap_title(title: str) -> str:
+ """Strip ``Revert "..."`` / ``Reapply "..."`` wrappers from a PR title.
+
+ ``main()`` keeps Revert and Reapply PR titles verbatim, so the JIRA id,
+ leading markers, and components all live inside the quotes. Unwrapping
here
+ lets the id-extraction and skip-tag logic see them. Wrappers are stripped
+ repeatedly, so a revert-of-revert (``Revert "Revert "...""``), including
+ mixed ``Revert``/``Reapply`` nesting, is reduced to its innermost title.
+ Surrounding whitespace is stripped from the title and from each unwrapped
+ layer. A title without such a wrapper is otherwise returned unchanged.
+
+ >>> _unwrap_title('Revert "[SPARK-1234][SQL] Fix"')
+ '[SPARK-1234][SQL] Fix'
+ >>> _unwrap_title('Reapply "[SPARK-1234][SQL] Fix"')
+ '[SPARK-1234][SQL] Fix'
+ >>> _unwrap_title('Revert "Revert "[SPARK-1234][SQL] Fix""')
+ '[SPARK-1234][SQL] Fix'
+ >>> _unwrap_title('Revert " Reapply " [SPARK-1234][SQL] Fix""')
+ '[SPARK-1234][SQL] Fix'
+ >>> _unwrap_title('[SPARK-1234][SQL] Fix')
+ '[SPARK-1234][SQL] Fix'
+ >>> _unwrap_title('Revert "')
+ 'Revert "'
+ """
+ title = title.strip()
+ while m := _WRAPPER_RE.fullmatch(title):
+ title = m.group(1).strip()
+ return title
+
+
+def leading_jira_ids(title: str) -> list[str]:
+ """Return the JIRA ids that open ``title``, in order, upper-cased, without
duplicates.
+
+ ``Revert "..."`` / ``Reapply "..."`` wrappers, including nested ones, are
+ stripped first (see :func:`_unwrap_title`), so those PRs still yield the
+ wrapped id. The ids are the run of ``[SPARK-N]`` bracket tags at the start
+ of the title. Scanning stops at the first other tag, so a SPARK id later
in
+ the title, such as a reference to a related ticket in the body, is not
+ collected. The tags after the ids are not validated, so a title that
+ :meth:`Title.parse` rejects, such as ``[SPARK-1234][SQL][MINOR] Fix``,
still
+ yields its id. A title that opens with a bare id instead of a bracket,
such
+ as the old subject ``SPARK-49004``, yields only that first id. Any other
+ title, including one whose id follows another tag, yields an empty list.
+
+ >>> leading_jira_ids("[SPARK-1234][SPARK-5678][SQL] Follow up to
SPARK-9999")
+ ['SPARK-1234', 'SPARK-5678']
+ >>> leading_jira_ids('Reapply "[SPARK-1234][SQL] Fix"')
+ ['SPARK-1234']
+ >>> leading_jira_ids('Revert "Revert "[SPARK-1234][SQL] Fix""')
+ ['SPARK-1234']
+ >>> leading_jira_ids("[SPARK-1234][spark-1234] Fix")
+ ['SPARK-1234']
+ >>> leading_jira_ids("[SPARK-123456][SQL] Fix")
+ ['SPARK-123456']
+ >>> leading_jira_ids("[SPARK-1234][SQL][MINOR] Fix")
+ ['SPARK-1234']
+ >>> leading_jira_ids('Revert " [SPARK-1234][SQL][MINOR] Fix"')
+ ['SPARK-1234']
+ >>> leading_jira_ids('Revert "SPARK-49004"')
+ ['SPARK-49004']
+ >>> leading_jira_ids('Revert "SPARK-9545, SPARK-9547: Fix"')
+ ['SPARK-9545']
+ >>> leading_jira_ids("[MINOR][SQL] Fix typo")
+ []
+ >>> leading_jira_ids("[WIP][SPARK-1234] Fix")
+ []
+ >>> leading_jira_ids("No leading tag, mentions SPARK-5678")
+ []
+ """
+ title = _unwrap_title(title)
+ m = _BARE_SPARK_ID_RE.match(title)
+ if m:
+ return [m.group(0).upper()]
+ ids = []
+ for tag, _ in _iter_bracket_tags(title):
+ if not _SPARK_ID_RE.match(tag):
+ break
+ ids.append(tag)
+ return list(dict.fromkeys(ids))
+
+
+def leading_title_tags(title: str) -> set[str]:
+ """Return the tags in the leading run of bracket tags of ``title``,
normalized.
+
+ ``Revert "..."`` / ``Reapply "..."`` wrappers are stripped first (see
+ :func:`_unwrap_title`), and aliases are normalized (see
+ :meth:`Component.normalize`). Like :func:`leading_jira_ids`, the run is
not
+ validated, so a title that :meth:`Title.parse` rejects still yields its
tags.
+
+ >>> sorted(leading_title_tags('Revert "[SPARK-1234][FOLLOW-UP][SQL][MINOR]
Fix [X]"'))
+ ['FOLLOWUP', 'MINOR', 'SPARK-1234', 'SQL']
+ >>> leading_title_tags("SPARK-49004")
+ set()
+ """
+ return {Component.normalize(tag) for tag, _ in
_iter_bracket_tags(_unwrap_title(title))}
+
+
def prompt_for_components():
"""
Prompt the committer for component(s) when the PR title lacks a primary
- component. Each entered token is normalized via Component.find
- (e.g. "DOCS" -> "DOC", "PYSPARK" -> "PYTHON"). Unrecognized tokens are
- passed through as-is. Re-prompts until at least one entered token resolves
- to a primary Component (one with primary=True). Returns an uppercase list
- of tags in insertion order.
+ component. Each entered token is normalized via Component.normalize
+ (e.g. "DOCS" -> "DOC", "PYSPARK" -> "PYTHON"); unrecognized tokens are only
+ upper-cased. Re-prompts until at least one entered token resolves to a
+ primary Component (one with primary=True) and every token is one that the
+ bracket-tag scan reads back from the title as a component. So a token with
+ other characters, such as "SPARK CORE", is rejected. A SPARK id is rejected
+ too: it belongs in the PR title. So are [MINOR] and [TRIVIAL]: they mark a
PR
+ without a JIRA ticket and belong at the start of its title, where
Title.parse
+ checks them. Returns an uppercase list of tags in insertion order.
"""
print("PR title is missing a primary [COMPONENT] tag.")
print("Primary components (one of these is required):")
@@ -2218,10 +2411,32 @@ def prompt_for_components():
c = Component.find(t)
if c is not None and c.primary:
has_primary = True
- components.append(c.tag if c else t)
+ components.append(Component.normalize(t))
if not components:
print_error("Component(s) cannot be empty. Please enter at least
one.")
continue
+ # A tag that the bracket scan cannot read would hide the tags after
it, such as
+ # FOLLOWUP, from the JIRA skip check, at this merge and at a later
revert. A SPARK id
+ # would become an extra linked ticket or an id after a component.
+ invalid = [
+ t
+ for t in components
+ if not _BRACKET_TAG_RE.fullmatch("[%s]" % t) or
_SPARK_ID_RE.match(t)
+ ]
+ if invalid:
+ print_error(
+ "Invalid component(s): %s. Use letters, digits, '.', '_', or
'-', and no SPARK id."
+ % ", ".join(invalid)
+ )
+ continue
+ markers = [t for t in components if t in _LEADING_TAGS]
+ if markers:
+ print_error(
+ "%s cannot be entered here. A PR without a JIRA ticket puts
[MINOR] or "
+ "[TRIVIAL] at the start of its title instead; a PR with one
cannot be marked so."
+ % ", ".join("[%s]" % t for t in markers)
+ )
+ continue
if not has_primary:
print_error(
"At least one component must be a primary tag (see list
above). "
@@ -2467,7 +2682,7 @@ def main():
c = Component.find(tag)
if c is not None and c.primary:
has_primary = True
- components.append(c.tag if c is not None else tag)
+ components.append(Component.normalize(tag))
if not has_primary:
new_tags = prompt_for_components()
components = new_tags + components
@@ -2618,63 +2833,7 @@ def main():
)
continue_maybe(msg)
- jira_ids = jira_ids_to_verify(title)
- # Epic / Umbrella tickets group related work and must not be resolved by a
single PR.
- # Collect every offender so the committer sees the full list in one shot
rather than
- # discovering them one-by-one across repeated merge attempts.
- blocking_issue_types = {"Epic", "Umbrella"}
- blockers = []
- linked_issues = []
- for jira_id in jira_ids:
- try:
- issue = asf_jira.issue(jira_id)
- except Exception:
- print_error("Unable to fetch summary of %s" % jira_id)
- continue
- print_jira_issue_summary(issue)
- linked_issues.append((jira_id, issue))
- issue_type = issue.fields.issuetype.name
- if issue_type in blocking_issue_types:
- blockers.append((jira_id, issue_type))
- if blockers:
- ids_str = ", ".join("%s (%s)" % (jid, t) for jid, t in blockers)
- fail(
- "Cannot merge PR #%s. Linked JIRA(s) %s are Umbrella or Epic "
- "tickets and MUST not be resolved by a single PR. File "
- "Sub-task(s) under %s and update the PR title to reference "
- "the Sub-task(s) instead." % (pr_num, ids_str, ids_str)
- )
-
- # Confirm each linked JIRA actually matches this PR before merging. A
committer can
- # reference the wrong ticket -- e.g. mistaking a GitHub PR number in the
body
- # ("Closes #NNNNN") for a SPARK JIRA id -- which resolves an unrelated
ticket on merge.
- # Show the PR title next to each ticket's summary, warn when a ticket is
already
- # Resolved/Closed (a fresh merge should target an open one), and require
confirmation.
- # For [FOLLOWUP] PRs the title describes the follow-up fix rather than the
ticket, so
- # its low similarity to the JIRA summary is expected and the warning is
suppressed.
- is_followup = "FOLLOWUP" in title_components
- for jira_id, issue in linked_issues:
- print()
- print(
- format_jira_verification(
- pr_num,
- title,
- jira_id,
- issue.fields.summary,
- issue.fields.status.name,
- issue.fields.issuetype.name,
- resolution=(
- issue.fields.resolution.name if issue.fields.resolution is
not None else None
- ),
- use_color=True,
- is_followup=is_followup,
- )
- )
- if get_input("Does %s match this PR? (y/N): " % jira_id, ["y", "n",
""]) != "y":
- fail(
- "Aborting: %s does not match PR #%s. Fix the PR title to
reference the "
- "correct JIRA ticket and retry." % (jira_id, pr_num)
- )
+ check_linked_jira_issues(pr_num, title)
print("\n=== Pull Request #%s ===" % pr_num)
print("title\t%s\nsource\t%s\ntarget\t%s\nurl\t%s" % (title, pr_repo_desc,
target_ref, url))
@@ -2886,23 +3045,23 @@ __test__ = {
... client.transition_issue.assert_not_called()
""",
"update_jira_for_revert": """
- Reverts update each referenced JIRA once and bypass the resolve/add path.
+ Reverts update each leading JIRA id once and bypass the resolve/add path.
>>> from contextlib import redirect_stdout
>>> from io import StringIO
>>> from unittest.mock import Mock, call, patch
>>> reverter, resolver, confirm = Mock(), Mock(), Mock()
- >>> def update(title, branches, is_revert=False):
+ >>> def update(title, branches, is_revert=False, components=()):
... reverter.reset_mock()
... resolver.reset_mock()
... confirm.reset_mock()
... with (
... patch.dict(update_jira_for_pr.__globals__,
revert_jira_issue=reverter,
- ... resolve_jira_issues=resolver,
continue_maybe=confirm),
+ ... resolve_jira_issue=resolver,
continue_maybe=confirm),
... redirect_stdout(StringIO()),
... ):
- ... update_jira_for_pr("123", title, branches, [],
is_revert=is_revert)
- >>> title = 'Revert "[SPARK-1234][SPARK-5678][SQL] Fix SPARK-1234"'
+ ... update_jira_for_pr("123", title, branches, list(components),
is_revert=is_revert)
+ >>> title = 'Revert "[SPARK-1234][SPARK-5678][SQL] Fix mentions
SPARK-9999"'
>>> update(title, ["master", "branch-4.x"], is_revert=True)
>>> assert reverter.call_args_list == [
... call(['master', 'branch-4.x'], 'SPARK-1234'),
@@ -2916,24 +3075,85 @@ __test__ = {
>>> update('Revert "Fix"', ["master"], is_revert=True)
>>> reverter.assert_called_once_with(["master"], "")
- Wrapped skip tags preserve the existing policy for MINOR, TRIVIAL, and
FOLLOWUP changes.
+ A bare leading id, as in the old subject SPARK-49004, still reaches the
revert path.
+
+ >>> update('Revert "SPARK-49004"', ["master"], is_revert=True)
+ >>> reverter.assert_called_once_with(["master"], "SPARK-49004")
+
+ Wrapped skip tags follow the same policy as direct merges for FOLLOWUP,
including the
+ FOLLOW-UP alias, and for MINOR and TRIVIAL when the title has no leading
id. The tags are
+ read even from a wrapped title that Title.parse rejects.
>>> for title in ('Revert "[MINOR][SQL] Fix"', 'Revert "[TRIVIAL][SQL]
Fix"',
- ... 'Revert "[SPARK-1234][SQL][FOLLOWUP] Fix"'):
+ ... 'Revert "[SPARK-1234][SQL][FOLLOWUP] Fix"',
+ ... 'Revert "[SPARK-1234][SQL][FOLLOW-UP] Fix"',
+ ... 'Revert "[SPARK-1234][SQL][FOLLOWUP][MINOR] Fix"',
+ ... 'Revert "[MINOR][TRIVIAL][SQL] Fix"', 'Revert
"[SQL][MINOR] Fix"',
+ ... 'Revert "[MINOR][SPARK-1234] Fix"'):
... update(title, ["master"], is_revert=True)
... reverter.assert_not_called()
... resolver.assert_not_called()
... confirm.assert_not_called()
- Ordinary merges and reapplies still resolve the issue or add fix versions.
+ Title.parse rejects an id next to MINOR or TRIVIAL, and so does the
component prompt, so
+ only an older subject opens with an id and has one. Such a commit was
normally merged as
+ the fix for its id, so the id is updated.
+
+ >>> update('Revert "[SPARK-1234][SQL][MINOR] Fix"', ["master"],
is_revert=True)
+ >>> reverter.assert_called_once_with(["master"], "SPARK-1234")
+
+ The skip tags come from the title alone, so direct merges skip through it.
+
+ >>> for title in ("[MINOR][SQL] Fix", "[TRIVIAL][SQL] Fix",
"[SPARK-1234][SQL][FOLLOWUP] Fix"):
+ ... update(title, ["master"], components=["SQL"])
+ ... reverter.assert_not_called()
+ ... resolver.assert_not_called()
+ ... confirm.assert_not_called()
+
+ Reapply titles are unwrapped too, so their skip tags apply as well.
- >>> for title in ('[SPARK-1234][SQL] Fix', 'Reapply "[SPARK-1234][SQL]
Fix"'):
+ >>> for title in ('Reapply "[MINOR][SQL] Fix"', 'Reapply
"[SPARK-1234][SQL][FOLLOWUP] Fix"',
+ ... 'Reapply "[SPARK-1234][SQL][FOLLOW-UP] Fix"',
+ ... 'Reapply "[SPARK-1234][SQL][FOLLOWUP][MINOR] Fix"'):
+ ... update(title, ["master"])
+ ... reverter.assert_not_called()
+ ... resolver.assert_not_called()
+ ... confirm.assert_not_called()
+
+ Nested wrappers are unwrapped before the skip-tag and id checks, so an
inner
+ skip tag is still honored and an inner id still reaches the revert path. A
+ revert of a reapply is used here because it is a revert both by its
outermost
+ wrapper, which is how main() sets is_revert today, and by wrapper nesting.
+
+ >>> update('Revert "Reapply "[MINOR][SQL] Fix""', ["master"],
is_revert=True)
+ >>> reverter.assert_not_called()
+ >>> resolver.assert_not_called()
+ >>> confirm.assert_not_called()
+ >>> update('Revert "Reapply "[SPARK-1234][SQL] Fix""', ["master"],
is_revert=True)
+ >>> reverter.assert_called_once_with(["master"], "SPARK-1234")
+
+ Ordinary merges and reapplies resolve the leading id, not an id in the
title body.
+
+ >>> for title in ('[SPARK-1234][SQL] Fix mentions SPARK-9999',
+ ... 'Reapply "[SPARK-1234][SQL] Fix"',
+ ... 'Reapply "[SPARK-1234][SQL][TRIVIAL] Fix"'):
... update(title, ["master"])
... reverter.assert_not_called()
... resolver.assert_called_once_with(
- ... title, ["master"],
+ ... ["master"],
... "Issue resolved by pull request
123\\n[https://github.com/apache/spark/pull/123]",
- ... [])
+ ... "SPARK-1234", title_components=[])
+ >>> update("[SPARK-1234][SPARK-5678][SQL] Fix", ["master"])
+ >>> [c.args[2] for c in resolver.call_args_list]
+ ['SPARK-1234', 'SPARK-5678']
+
+ A title without an id still offers to resolve one that the committer
enters.
+
+ >>> update('Reapply "Fix"', ["master"])
+ >>> resolver.assert_called_once_with(
+ ... ["master"],
+ ... "Issue resolved by pull request
123\\n[https://github.com/apache/spark/pull/123]",
+ ... title_components=[])
When JIRA was disabled at startup (unreachable), every shape of update is
skipped: no
prompt, no resolve, no revert. The disabled message is the one printed
even for titles
@@ -2946,7 +3166,7 @@ __test__ = {
... output = StringIO()
... with (
... patch.dict(update_jira_for_pr.__globals__,
revert_jira_issue=reverter,
- ... resolve_jira_issues=resolver,
continue_maybe=confirm,
+ ... resolve_jira_issue=resolver, continue_maybe=confirm,
... jira_disabled=True),
... redirect_stdout(output),
... ):
@@ -2964,6 +3184,142 @@ __test__ = {
>>> all("because title has" not in o for o in disabled_outputs)
True
""",
+ "prompt_for_components": """
+ The prompt rejects MINOR and TRIVIAL, which belong at the start of the
title, tokens that
+ the bracket-tag scan cannot read back, and SPARK ids, and asks again until
the components
+ include a primary one.
+
+ >>> from contextlib import redirect_stdout
+ >>> from io import StringIO
+ >>> from unittest.mock import patch
+ >>> output = StringIO()
+ >>> answers = ["sql,minor", " Trivial ", "sql, spark core", "sql,[tests]",
"sql,spark-2",
+ ... "sql][minor", "tests", "sql, follow-up"]
+ >>> with (
+ ... patch.dict(prompt_for_components.__globals__, bold_input=lambda _:
answers.pop(0)),
+ ... redirect_stdout(output),
+ ... ):
+ ... components = prompt_for_components()
+ >>> components
+ ['SQL', 'FOLLOWUP']
+ >>> answers
+ []
+ >>> "[MINOR] cannot be entered here" in output.getvalue()
+ True
+ >>> "[TRIVIAL] cannot be entered here" in output.getvalue()
+ True
+ >>> for bad in ("SPARK CORE", "[TESTS]", "SPARK-2", "SQL][MINOR"):
+ ... assert "Invalid component(s): %s." % bad in output.getvalue(), bad
+ """,
+ "check_linked_jira_issues": """
+ Only the leading ids are fetched and checked, so an Epic that the title
body mentions
+ neither blocks the merge nor asks for confirmation.
+
+ >>> from contextlib import redirect_stdout
+ >>> from io import StringIO
+ >>> from unittest.mock import Mock, call, patch
+ >>> def issue(issue_type):
+ ... i = Mock()
+ ... i.fields.issuetype.name = issue_type
+ ... return i
+ >>> client = Mock()
+ >>> client.issue.side_effect = {
+ ... "SPARK-60328": issue("Sub-task"), "SPARK-60329": issue("Bug"),
+ ... "SPARK-34261": issue("Epic"), "SPARK-34262":
issue("Umbrella")}.__getitem__
+ >>> verify = Mock(return_value="")
+ >>> stop = Mock(side_effect=SystemExit)
+ >>> def check(title, answers=("y",)):
+ ... for m in (client, verify, stop):
+ ... m.reset_mock()
+ ... with (
+ ... patch.dict(check_linked_jira_issues.__globals__,
asf_jira=client,
+ ... print_jira_issue_summary=Mock(),
format_jira_verification=verify,
+ ... fail=stop),
+ ... patch("builtins.input", side_effect=answers),
+ ... redirect_stdout(StringIO()),
+ ... ):
+ ... check_linked_jira_issues("123", title)
+ >>> check("[SPARK-60328][SQL][TESTS] Use a unique jar copy in the
HiveDDLSuite "
+ ... "SPARK-34261 test")
+ >>> client.issue.call_args_list
+ [call('SPARK-60328')]
+ >>> stop.assert_not_called()
+ >>> verify.call_args.args[2]
+ 'SPARK-60328'
+
+ Each leading id is confirmed, and declining one aborts the merge.
+
+ >>> check("[SPARK-60328][SPARK-60329][SQL] Fix", answers=("y", "y"))
+ >>> [c.args[2] for c in verify.call_args_list]
+ ['SPARK-60328', 'SPARK-60329']
+ >>> stop.assert_not_called()
+ >>> try:
+ ... check("[SPARK-60328][SQL] Fix", answers=("n",))
+ ... except SystemExit:
+ ... pass
+ >>> stop.call_args.args[0].startswith("Aborting: SPARK-60328 does not
match PR #123.")
+ True
+ >>> try:
+ ... check("[SPARK-60328][SQL] Fix", answers=("",))
+ ... except SystemExit:
+ ... pass
+ >>> stop.call_count
+ 1
+
+ A ticket that cannot be fetched is skipped.
+
+ >>> check("[SPARK-1][SQL] Fix")
+ >>> client.issue.call_args_list
+ [call('SPARK-1')]
+ >>> verify.assert_not_called()
+ >>> stop.assert_not_called()
+ >>> try:
+ ... check("[SPARK-1][SPARK-34261][SQL] Fix")
+ ... except SystemExit:
+ ... pass
+ >>> "SPARK-34261 (Epic)" in stop.call_args.args[0]
+ True
+
+ When JIRA was disabled for this run, nothing is fetched or confirmed.
+
+ >>> client.reset_mock()
+ >>> output = StringIO()
+ >>> with (
+ ... patch.dict(check_linked_jira_issues.__globals__, asf_jira=client,
jira_disabled=True),
+ ... redirect_stdout(output),
+ ... ):
+ ... check_linked_jira_issues("123", "[SPARK-60328][SQL] Fix")
+ >>> client.issue.assert_not_called()
+ >>> "skipping the Epic/Umbrella check" in output.getvalue()
+ True
+
+ Leading Epic and Umbrella tickets block the merge, all listed at once,
before any
+ confirmation.
+
+ >>> try:
+ ... check("[SPARK-34261][SPARK-60328][SPARK-34262][SQL] Fix")
+ ... except SystemExit:
+ ... pass
+ >>> msg = stop.call_args.args[0]
+ >>> "SPARK-34261 (Epic), SPARK-34262 (Umbrella)" in msg
+ True
+ >>> verify.assert_not_called()
+
+ A FOLLOWUP tag suppresses the similarity check inside Revert/Reapply
wrappers and
+ under the FOLLOW-UP alias too, while a title without the tag is still
checked.
+
+ >>> for title in ('[SPARK-60328][SQL][FOLLOWUP] Fix', 'Revert
"[SPARK-60328][SQL] Fix"',
+ ... '[SPARK-60328][SQL] Fix FOLLOWUP handling',
+ ... 'Revert "[SPARK-60328][SQL][FOLLOW-UP] Fix"',
+ ... 'Reapply "[SPARK-60328][FOLLOWUP][SQL][MINOR] Fix"'):
+ ... check(title)
+ ... print(verify.call_args.kwargs["is_followup"])
+ True
+ False
+ False
+ True
+ True
+ """,
"reconcile_jira_affects_versions": """
Blank input accepts the suggested version and appends it to the existing
versions:
@@ -3048,7 +3404,7 @@ __test__ = {
True
""",
"jira_ids_to_verify": """
- The ids from the title feed main()'s Epic/Umbrella and match checks; with
JIRA disabled
+ The leading ids of the title feed the Epic/Umbrella and match checks; with
JIRA disabled
they are dropped -- with a reminder naming them, or the generic fallback
-- so the checks
are skipped rather than failing on asf_jira being None.
@@ -3064,7 +3420,7 @@ __test__ = {
... return jira_ids_to_verify(title), output.getvalue()
>>> ids_for("[SPARK-1234][SPARK-5678][SQL] Fix", False)
(['SPARK-1234', 'SPARK-5678'], '')
- >>> ids, output = ids_for("[SPARK-1234][SQL] Fix", True)
+ >>> ids, output = ids_for("[SPARK-1234][SQL] Fix mentions SPARK-9999",
True)
>>> ids, "Confirm SPARK-1234 manually" in output
([], True)
>>> ids, output = ids_for("[SQL] Fix", True)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]