Copilot commented on code in PR #13063: URL: https://github.com/apache/trafficserver/pull/13063#discussion_r3970948088
########## tools/changelog/changelog.py: ########## @@ -0,0 +1,509 @@ +#!/usr/bin/env python3 +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +"""Generate a changelog from merged PRs in a GitHub milestone. + +Usage: + uv run --project tools/changelog python tools/changelog/changelog.py \ + -o apache -r trafficserver -m 10.2.0 + +Output is written to stdout in the format used by CHANGELOG-* files: + + Changes with Apache Traffic Server 10.2.0 + #11945 - Make directory operations methods on `Directory` + #12026 - Static link opentelemetry-cpp libraries to otel_tracer plugin + ... + +To generate a changelog file for a release: + + uv run --project tools/changelog python tools/changelog/changelog.py \ + -o apache -r trafficserver -m 10.2.0 > CHANGELOG-10.2.0 + +Use --doc to include extra metadata (merge commit SHA, PR body, labels) for each +PR, useful for generating release documentation. With --from-git the body is the +commit message body rather than a PR body: + + uv run --project tools/changelog python tools/changelog/changelog.py \ + -o apache -r trafficserver -m 10.2.0 --doc --format yaml > changelog.yaml + +For security releases whose fixes land directly on a release branch without +public pull requests (and therefore never appear in a milestone), use +--from-git to source the changelog from a git commit range. This is merged +with the milestone PRs (deduplicated by PR number) so the result includes both +the direct-to-branch commits and any milestone PRs. The -m value is still used +as the version label in the output: + + uv run --project tools/changelog python tools/changelog/changelog.py \ + -o apache -r trafficserver -m 10.1.4 \ + --from-git 10.1.3..upstream/10.1.x + +Requires a GitHub token via GH_TOKEN env var or -a flag to avoid rate limits. +""" + +import argparse +import json +import os +import re +import subprocess +import sys + +import httpx + +try: + import yaml +except ImportError: + yaml = None + Review Comment: The code treats PyYAML as an optional dependency (graceful ImportError + runtime error on `--format yaml`), but the project metadata currently declares `pyyaml>=6.0` as a required dependency. To keep behavior and packaging consistent, either (a) make PyYAML truly optional (declare it as an extra / optional dependency and keep the ImportError path), or (b) make it required in code (remove the try/except and the conditional runtime error since it should never trigger). ########## tools/changelog/changelog.py: ########## @@ -0,0 +1,509 @@ +#!/usr/bin/env python3 +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +"""Generate a changelog from merged PRs in a GitHub milestone. + +Usage: + uv run --project tools/changelog python tools/changelog/changelog.py \ + -o apache -r trafficserver -m 10.2.0 + +Output is written to stdout in the format used by CHANGELOG-* files: + + Changes with Apache Traffic Server 10.2.0 + #11945 - Make directory operations methods on `Directory` + #12026 - Static link opentelemetry-cpp libraries to otel_tracer plugin + ... + +To generate a changelog file for a release: + + uv run --project tools/changelog python tools/changelog/changelog.py \ + -o apache -r trafficserver -m 10.2.0 > CHANGELOG-10.2.0 + +Use --doc to include extra metadata (merge commit SHA, PR body, labels) for each +PR, useful for generating release documentation. With --from-git the body is the +commit message body rather than a PR body: + + uv run --project tools/changelog python tools/changelog/changelog.py \ + -o apache -r trafficserver -m 10.2.0 --doc --format yaml > changelog.yaml + +For security releases whose fixes land directly on a release branch without +public pull requests (and therefore never appear in a milestone), use +--from-git to source the changelog from a git commit range. This is merged +with the milestone PRs (deduplicated by PR number) so the result includes both +the direct-to-branch commits and any milestone PRs. The -m value is still used +as the version label in the output: + + uv run --project tools/changelog python tools/changelog/changelog.py \ + -o apache -r trafficserver -m 10.1.4 \ + --from-git 10.1.3..upstream/10.1.x + +Requires a GitHub token via GH_TOKEN env var or -a flag to avoid rate limits. +""" + +import argparse +import json +import os +import re +import subprocess +import sys + +import httpx + +try: + import yaml +except ImportError: + yaml = None + +API_URL = "https://api.github.com"; + + +def gh_cli_available() -> bool: + try: + subprocess.run(["gh", "--version"], capture_output=True, check=True) + return True + except (FileNotFoundError, subprocess.CalledProcessError): + return False + + +def changelog_via_gh(owner: str, repo: str, milestone: str, verbose: bool, doc: bool) -> list[dict]: + """Use the gh CLI to fetch milestone PRs (avoids API rate limits).""" + milestone_id = None + result = subprocess.run( + ["gh", "api", f"/repos/{owner}/{repo}/milestones?state=all", "--paginate"], + capture_output=True, + text=True, + ) + if result.returncode != 0: + print(f"gh api error: {result.stderr}", file=sys.stderr) + sys.exit(1) + + milestones = json.loads(result.stdout) + for ms in milestones: + if ms["title"] == milestone: + milestone_id = ms["number"] + break + + if milestone_id is None: + print(f"Milestone not found: {milestone}", file=sys.stderr) + sys.exit(1) + + print(f"Looking for issues from Milestone {milestone}", file=sys.stderr) + + changelog = [] + page = 1 + while True: + print(f"Page {page}", file=sys.stderr) + result = subprocess.run( + [ + "gh", + "api", + f"/repos/{owner}/{repo}/issues?milestone={milestone_id}&state=closed&page={page}&per_page=100", + ], + capture_output=True, + text=True, + ) + if result.returncode != 0: + print(f"gh api error: {result.stderr}", file=sys.stderr) + sys.exit(1) + + issues = json.loads(result.stdout) + if not issues: + break + + for issue in issues: + number = issue["number"] + title = issue["title"] + if verbose: + print(f"Issue #{number} - {title} ", end="", file=sys.stderr) + + if "pull_request" not in issue: + if verbose: + print("not a PR.", file=sys.stderr) + continue + + if not _gh_is_merged(owner, repo, number): + if verbose: + print("not merged.", file=sys.stderr) + continue + + if verbose: + print("added.", file=sys.stderr) + + entry: dict = {"number": number, "title": title} + if doc: + labels = [label["name"] for label in issue.get("labels", [])] + entry["labels"] = labels + pr_detail = subprocess.run( + ["gh", "api", f"/repos/{owner}/{repo}/pulls/{number}"], + capture_output=True, + text=True, + ) + if pr_detail.returncode != 0: + print(f"gh api error fetching PR #{number}: {pr_detail.stderr.strip()}", file=sys.stderr) + sys.exit(1) + pr_data = json.loads(pr_detail.stdout) + entry["sha"] = pr_data.get("merge_commit_sha", "") + entry["body"] = pr_data.get("body", "") or "" + changelog.append(entry) + + page += 1 + + return changelog + + +def changelog_via_api( + owner: str, + repo: str, + milestone: str, + token: str | None, + verbose: bool, + doc: bool, +) -> list[dict]: + """Use httpx to call the GitHub REST API directly.""" + headers = { + "Accept": "application/vnd.github.v3+json", + "User-Agent": "ATS-Changelog-Tool", + } + if token: + headers["Authorization"] = f"Bearer {token}" + + with httpx.Client(base_url=API_URL, headers=headers, timeout=30) as client: + milestone_id = _lookup_milestone(client, owner, repo, milestone) + if milestone_id is None: + print(f"Milestone not found: {milestone}", file=sys.stderr) + sys.exit(1) + + print(f"Looking for issues from Milestone {milestone}", file=sys.stderr) + + changelog = [] + page = 1 + while True: + print(f"Page {page}", file=sys.stderr) + resp = client.get( + f"/repos/{owner}/{repo}/issues", + params={ + "milestone": milestone_id, + "state": "closed", + "page": page, + "per_page": 100, + }, + ) + _check_rate_limit(resp) + resp.raise_for_status() + issues = resp.json() + + if not issues: + break + + for issue in issues: + number = issue["number"] + title = issue["title"] + if verbose: + print(f"Issue #{number} - {title} ", end="", file=sys.stderr) + + if "pull_request" not in issue: + if verbose: + print("not a PR.", file=sys.stderr) + continue + + if not _is_merged(client, owner, repo, number): + if verbose: + print("not merged.", file=sys.stderr) + continue + + if verbose: + print("added.", file=sys.stderr) + + entry: dict = {"number": number, "title": title} + if doc: + labels = [label["name"] for label in issue.get("labels", [])] + entry["labels"] = labels + pr_resp = client.get(f"/repos/{owner}/{repo}/pulls/{number}") + _check_rate_limit(pr_resp) + pr_resp.raise_for_status() + pr_data = pr_resp.json() + entry["sha"] = pr_data.get("merge_commit_sha", "") + entry["body"] = pr_data.get("body", "") or "" + changelog.append(entry) + + page += 1 + + return changelog + + +def changelog_via_git(git_range: str, verbose: bool, doc: bool) -> list[dict]: + """Build the changelog from a git commit range. + + Used for security releases whose fixes are committed directly to a release + branch without public PRs, so they never appear in a GitHub milestone. Each + commit becomes an entry; a trailing "(#N)" in the subject is captured as the + PR number when present, but is not required. + """ + field_sep = "\x1f" + record_sep = "\x1e" + fmt = f"%H{field_sep}%s{field_sep}%b{record_sep}" + result = subprocess.run( + ["git", "log", "--no-merges", "--reverse", f"--pretty=format:{fmt}", git_range], + capture_output=True, + text=True, + ) + if result.returncode != 0: + print(f"git log error: {result.stderr.strip()}", file=sys.stderr) + sys.exit(1) + + print(f"Reading commits from git range {git_range}", file=sys.stderr) + + changelog = [] + for record in result.stdout.split(record_sep): + record = record.strip("\n") + if not record: + continue + sha, subject, body = record.split(field_sep, 2) + # Only a trailing "(#N)" is a PR number. A bare "#N" anywhere in the subject + # is usually an issue reference, and treating it as a PR number would merge + # unrelated entries in merge_changelogs(). + match = re.search(r"\(#(\d+)\)$", subject) + number = int(match.group(1)) if match else None + if verbose: + label = f"#{number}" if number else sha[:12] + print(f"{label} - {subject} added.", file=sys.stderr) + + entry: dict = {"number": number, "title": subject} + if doc: + entry["sha"] = sha + entry["body"] = body.strip() + changelog.append(entry) + + return changelog + + +def merge_changelogs(milestone_entries: list[dict], git_entries: list[dict]) -> list[dict]: + """Union the milestone-sourced and git-sourced entries, deduplicated by PR number. + + Git commits keep their chronological order and carry commit-derived metadata. + Where a git commit's PR number matches a milestone PR, the milestone labels are + grafted onto the git entry. Milestone PRs with no corresponding commit in the + git range (e.g. folded into a combined backport) are appended, sorted by number. + """ + ms_by_number = {e["number"]: e for e in milestone_entries} + git_numbers = {e["number"] for e in git_entries if e.get("number")} + + for ge in git_entries: + num = ge.get("number") + if num in ms_by_number and ms_by_number[num].get("labels") and "labels" not in ge: + ge["labels"] = ms_by_number[num]["labels"] + + extras = [e for e in milestone_entries if e["number"] not in git_numbers] + extras.sort(key=lambda x: x["number"]) + + return git_entries + extras Review Comment: `merge_changelogs()`’s docstring says entries are “deduplicated by PR number”, but the implementation only avoids duplicates between milestone entries and git-sourced entries; it does not dedupe duplicates within `git_entries` itself. If multiple commits in the git range end with the same `(#N)`, you’ll emit multiple entries for the same PR number (and still append milestone extras based on a set), which contradicts the documented behavior. Consider tracking `seen_numbers` while building the returned list (keeping the first occurrence to preserve chronological order, or merging bodies/SHAs deterministically) so the output truly deduplicates by PR number. ########## tools/changelog/changelog.py: ########## @@ -0,0 +1,509 @@ +#!/usr/bin/env python3 +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +"""Generate a changelog from merged PRs in a GitHub milestone. + +Usage: + uv run --project tools/changelog python tools/changelog/changelog.py \ + -o apache -r trafficserver -m 10.2.0 + +Output is written to stdout in the format used by CHANGELOG-* files: + + Changes with Apache Traffic Server 10.2.0 + #11945 - Make directory operations methods on `Directory` + #12026 - Static link opentelemetry-cpp libraries to otel_tracer plugin + ... + +To generate a changelog file for a release: + + uv run --project tools/changelog python tools/changelog/changelog.py \ + -o apache -r trafficserver -m 10.2.0 > CHANGELOG-10.2.0 + +Use --doc to include extra metadata (merge commit SHA, PR body, labels) for each +PR, useful for generating release documentation. With --from-git the body is the +commit message body rather than a PR body: + + uv run --project tools/changelog python tools/changelog/changelog.py \ + -o apache -r trafficserver -m 10.2.0 --doc --format yaml > changelog.yaml + +For security releases whose fixes land directly on a release branch without +public pull requests (and therefore never appear in a milestone), use +--from-git to source the changelog from a git commit range. This is merged +with the milestone PRs (deduplicated by PR number) so the result includes both +the direct-to-branch commits and any milestone PRs. The -m value is still used +as the version label in the output: + + uv run --project tools/changelog python tools/changelog/changelog.py \ + -o apache -r trafficserver -m 10.1.4 \ + --from-git 10.1.3..upstream/10.1.x + +Requires a GitHub token via GH_TOKEN env var or -a flag to avoid rate limits. +""" + +import argparse +import json +import os +import re +import subprocess +import sys + +import httpx + +try: + import yaml +except ImportError: + yaml = None + +API_URL = "https://api.github.com"; + + +def gh_cli_available() -> bool: + try: + subprocess.run(["gh", "--version"], capture_output=True, check=True) + return True + except (FileNotFoundError, subprocess.CalledProcessError): + return False + + +def changelog_via_gh(owner: str, repo: str, milestone: str, verbose: bool, doc: bool) -> list[dict]: + """Use the gh CLI to fetch milestone PRs (avoids API rate limits).""" + milestone_id = None + result = subprocess.run( + ["gh", "api", f"/repos/{owner}/{repo}/milestones?state=all", "--paginate"], + capture_output=True, + text=True, + ) + if result.returncode != 0: + print(f"gh api error: {result.stderr}", file=sys.stderr) + sys.exit(1) + + milestones = json.loads(result.stdout) + for ms in milestones: + if ms["title"] == milestone: + milestone_id = ms["number"] + break + + if milestone_id is None: + print(f"Milestone not found: {milestone}", file=sys.stderr) + sys.exit(1) + + print(f"Looking for issues from Milestone {milestone}", file=sys.stderr) + + changelog = [] + page = 1 + while True: + print(f"Page {page}", file=sys.stderr) + result = subprocess.run( + [ + "gh", + "api", + f"/repos/{owner}/{repo}/issues?milestone={milestone_id}&state=closed&page={page}&per_page=100", + ], + capture_output=True, + text=True, + ) + if result.returncode != 0: + print(f"gh api error: {result.stderr}", file=sys.stderr) + sys.exit(1) + + issues = json.loads(result.stdout) + if not issues: + break + + for issue in issues: + number = issue["number"] + title = issue["title"] + if verbose: + print(f"Issue #{number} - {title} ", end="", file=sys.stderr) + + if "pull_request" not in issue: + if verbose: + print("not a PR.", file=sys.stderr) + continue + + if not _gh_is_merged(owner, repo, number): + if verbose: + print("not merged.", file=sys.stderr) + continue + + if verbose: + print("added.", file=sys.stderr) + + entry: dict = {"number": number, "title": title} + if doc: + labels = [label["name"] for label in issue.get("labels", [])] + entry["labels"] = labels + pr_detail = subprocess.run( + ["gh", "api", f"/repos/{owner}/{repo}/pulls/{number}"], + capture_output=True, + text=True, + ) + if pr_detail.returncode != 0: + print(f"gh api error fetching PR #{number}: {pr_detail.stderr.strip()}", file=sys.stderr) + sys.exit(1) + pr_data = json.loads(pr_detail.stdout) + entry["sha"] = pr_data.get("merge_commit_sha", "") + entry["body"] = pr_data.get("body", "") or "" + changelog.append(entry) + + page += 1 + + return changelog + + +def changelog_via_api( + owner: str, + repo: str, + milestone: str, + token: str | None, + verbose: bool, + doc: bool, +) -> list[dict]: + """Use httpx to call the GitHub REST API directly.""" + headers = { + "Accept": "application/vnd.github.v3+json", + "User-Agent": "ATS-Changelog-Tool", + } + if token: + headers["Authorization"] = f"Bearer {token}" + + with httpx.Client(base_url=API_URL, headers=headers, timeout=30) as client: + milestone_id = _lookup_milestone(client, owner, repo, milestone) + if milestone_id is None: + print(f"Milestone not found: {milestone}", file=sys.stderr) + sys.exit(1) + + print(f"Looking for issues from Milestone {milestone}", file=sys.stderr) + + changelog = [] + page = 1 + while True: + print(f"Page {page}", file=sys.stderr) + resp = client.get( + f"/repos/{owner}/{repo}/issues", + params={ + "milestone": milestone_id, + "state": "closed", + "page": page, + "per_page": 100, + }, + ) + _check_rate_limit(resp) + resp.raise_for_status() + issues = resp.json() + + if not issues: + break + + for issue in issues: + number = issue["number"] + title = issue["title"] + if verbose: + print(f"Issue #{number} - {title} ", end="", file=sys.stderr) + + if "pull_request" not in issue: + if verbose: + print("not a PR.", file=sys.stderr) + continue + + if not _is_merged(client, owner, repo, number): + if verbose: + print("not merged.", file=sys.stderr) + continue + + if verbose: + print("added.", file=sys.stderr) + + entry: dict = {"number": number, "title": title} + if doc: + labels = [label["name"] for label in issue.get("labels", [])] + entry["labels"] = labels + pr_resp = client.get(f"/repos/{owner}/{repo}/pulls/{number}") + _check_rate_limit(pr_resp) + pr_resp.raise_for_status() + pr_data = pr_resp.json() + entry["sha"] = pr_data.get("merge_commit_sha", "") + entry["body"] = pr_data.get("body", "") or "" + changelog.append(entry) Review Comment: In `--doc` mode, each PR incurs two API calls: one to `/pulls/{n}/merge` (via `_is_merged`) and another to `/pulls/{n}` to fetch metadata. Since the PR detail response includes merge state (`merged_at` / `merged` depending on API shape), you can avoid the extra merge-check call when `doc=True` by fetching PR details once and determining merged status from that response. This reduces API usage and lowers the likelihood of rate limiting/timeouts for large milestones. ########## tools/changelog/pyproject.toml: ########## @@ -0,0 +1,29 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +[project] +name = "changelog" +version = "0.1.0" +description = "Generate changelog from GitHub milestones for Apache Traffic Server" +requires-python = ">=3.11" +license = "Apache-2.0" +dependencies = [ + "httpx>=0.27", Review Comment: `pyyaml` is listed as a mandatory dependency here, but the implementation gates YAML output behind a conditional import and prints an installation instruction if it’s missing. If the intent is to support YAML output only when requested, consider moving `pyyaml` to an optional extra (e.g., `changelog[yaml]`) so minimal installs don’t pull it in unnecessarily; otherwise, if YAML is always supported, align the code by treating PyYAML as required (no optional import / runtime install hint). ########## doc/developer-guide/release-process/index.en.rst: ########## @@ -67,8 +67,8 @@ Build #. Generate or update the CHANGELOG for the next release. :: - ./tools/git/changelog.pl -o apache -r trafficserver -m X.Y.Z > - CHANGELOG-X.Y.Z + uv run --project tools/changelog python tools/changelog/changelog.py \ Review Comment: `tools/changelog/pyproject.toml` defines a console script entry point (`changelog = "changelog:main"`), but the docs invoke the tool via `python tools/changelog/changelog.py`. Using the console script (e.g., `uv run --project tools/changelog changelog ...`) would better reflect the supported interface, keep the documentation stable if file paths change, and avoid relying on module file location. ########## tools/changelog/changelog.py: ########## @@ -0,0 +1,509 @@ +#!/usr/bin/env python3 +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +"""Generate a changelog from merged PRs in a GitHub milestone. + +Usage: + uv run --project tools/changelog python tools/changelog/changelog.py \ + -o apache -r trafficserver -m 10.2.0 + +Output is written to stdout in the format used by CHANGELOG-* files: + + Changes with Apache Traffic Server 10.2.0 + #11945 - Make directory operations methods on `Directory` + #12026 - Static link opentelemetry-cpp libraries to otel_tracer plugin + ... + +To generate a changelog file for a release: + + uv run --project tools/changelog python tools/changelog/changelog.py \ + -o apache -r trafficserver -m 10.2.0 > CHANGELOG-10.2.0 + +Use --doc to include extra metadata (merge commit SHA, PR body, labels) for each +PR, useful for generating release documentation. With --from-git the body is the +commit message body rather than a PR body: + + uv run --project tools/changelog python tools/changelog/changelog.py \ + -o apache -r trafficserver -m 10.2.0 --doc --format yaml > changelog.yaml + +For security releases whose fixes land directly on a release branch without +public pull requests (and therefore never appear in a milestone), use +--from-git to source the changelog from a git commit range. This is merged +with the milestone PRs (deduplicated by PR number) so the result includes both +the direct-to-branch commits and any milestone PRs. The -m value is still used +as the version label in the output: + + uv run --project tools/changelog python tools/changelog/changelog.py \ + -o apache -r trafficserver -m 10.1.4 \ + --from-git 10.1.3..upstream/10.1.x + +Requires a GitHub token via GH_TOKEN env var or -a flag to avoid rate limits. +""" + +import argparse +import json +import os +import re +import subprocess +import sys + +import httpx + +try: + import yaml +except ImportError: + yaml = None + +API_URL = "https://api.github.com"; + + +def gh_cli_available() -> bool: + try: + subprocess.run(["gh", "--version"], capture_output=True, check=True) + return True + except (FileNotFoundError, subprocess.CalledProcessError): + return False + + +def changelog_via_gh(owner: str, repo: str, milestone: str, verbose: bool, doc: bool) -> list[dict]: + """Use the gh CLI to fetch milestone PRs (avoids API rate limits).""" + milestone_id = None + result = subprocess.run( + ["gh", "api", f"/repos/{owner}/{repo}/milestones?state=all", "--paginate"], + capture_output=True, + text=True, + ) + if result.returncode != 0: + print(f"gh api error: {result.stderr}", file=sys.stderr) + sys.exit(1) + + milestones = json.loads(result.stdout) + for ms in milestones: + if ms["title"] == milestone: + milestone_id = ms["number"] + break + + if milestone_id is None: + print(f"Milestone not found: {milestone}", file=sys.stderr) + sys.exit(1) + + print(f"Looking for issues from Milestone {milestone}", file=sys.stderr) + + changelog = [] + page = 1 + while True: + print(f"Page {page}", file=sys.stderr) + result = subprocess.run( + [ + "gh", + "api", + f"/repos/{owner}/{repo}/issues?milestone={milestone_id}&state=closed&page={page}&per_page=100", + ], + capture_output=True, + text=True, + ) + if result.returncode != 0: + print(f"gh api error: {result.stderr}", file=sys.stderr) + sys.exit(1) + + issues = json.loads(result.stdout) + if not issues: + break + + for issue in issues: + number = issue["number"] + title = issue["title"] + if verbose: + print(f"Issue #{number} - {title} ", end="", file=sys.stderr) + + if "pull_request" not in issue: + if verbose: + print("not a PR.", file=sys.stderr) + continue + + if not _gh_is_merged(owner, repo, number): + if verbose: + print("not merged.", file=sys.stderr) + continue + + if verbose: + print("added.", file=sys.stderr) + + entry: dict = {"number": number, "title": title} + if doc: + labels = [label["name"] for label in issue.get("labels", [])] + entry["labels"] = labels + pr_detail = subprocess.run( + ["gh", "api", f"/repos/{owner}/{repo}/pulls/{number}"], + capture_output=True, + text=True, + ) + if pr_detail.returncode != 0: + print(f"gh api error fetching PR #{number}: {pr_detail.stderr.strip()}", file=sys.stderr) + sys.exit(1) + pr_data = json.loads(pr_detail.stdout) + entry["sha"] = pr_data.get("merge_commit_sha", "") + entry["body"] = pr_data.get("body", "") or "" + changelog.append(entry) + + page += 1 + + return changelog + + +def changelog_via_api( + owner: str, + repo: str, + milestone: str, + token: str | None, + verbose: bool, + doc: bool, +) -> list[dict]: + """Use httpx to call the GitHub REST API directly.""" + headers = { + "Accept": "application/vnd.github.v3+json", + "User-Agent": "ATS-Changelog-Tool", + } + if token: + headers["Authorization"] = f"Bearer {token}" + + with httpx.Client(base_url=API_URL, headers=headers, timeout=30) as client: + milestone_id = _lookup_milestone(client, owner, repo, milestone) + if milestone_id is None: + print(f"Milestone not found: {milestone}", file=sys.stderr) + sys.exit(1) + + print(f"Looking for issues from Milestone {milestone}", file=sys.stderr) + + changelog = [] + page = 1 + while True: + print(f"Page {page}", file=sys.stderr) + resp = client.get( + f"/repos/{owner}/{repo}/issues", + params={ + "milestone": milestone_id, + "state": "closed", + "page": page, + "per_page": 100, + }, + ) + _check_rate_limit(resp) + resp.raise_for_status() + issues = resp.json() + + if not issues: + break + + for issue in issues: + number = issue["number"] + title = issue["title"] + if verbose: + print(f"Issue #{number} - {title} ", end="", file=sys.stderr) + + if "pull_request" not in issue: + if verbose: + print("not a PR.", file=sys.stderr) + continue + + if not _is_merged(client, owner, repo, number): + if verbose: + print("not merged.", file=sys.stderr) + continue + + if verbose: + print("added.", file=sys.stderr) + + entry: dict = {"number": number, "title": title} + if doc: + labels = [label["name"] for label in issue.get("labels", [])] + entry["labels"] = labels + pr_resp = client.get(f"/repos/{owner}/{repo}/pulls/{number}") + _check_rate_limit(pr_resp) + pr_resp.raise_for_status() + pr_data = pr_resp.json() + entry["sha"] = pr_data.get("merge_commit_sha", "") + entry["body"] = pr_data.get("body", "") or "" + changelog.append(entry) + + page += 1 + + return changelog + + +def changelog_via_git(git_range: str, verbose: bool, doc: bool) -> list[dict]: + """Build the changelog from a git commit range. + + Used for security releases whose fixes are committed directly to a release + branch without public PRs, so they never appear in a GitHub milestone. Each + commit becomes an entry; a trailing "(#N)" in the subject is captured as the + PR number when present, but is not required. + """ + field_sep = "\x1f" + record_sep = "\x1e" + fmt = f"%H{field_sep}%s{field_sep}%b{record_sep}" + result = subprocess.run( + ["git", "log", "--no-merges", "--reverse", f"--pretty=format:{fmt}", git_range], + capture_output=True, + text=True, + ) + if result.returncode != 0: + print(f"git log error: {result.stderr.strip()}", file=sys.stderr) + sys.exit(1) + + print(f"Reading commits from git range {git_range}", file=sys.stderr) + + changelog = [] + for record in result.stdout.split(record_sep): + record = record.strip("\n") + if not record: + continue + sha, subject, body = record.split(field_sep, 2) + # Only a trailing "(#N)" is a PR number. A bare "#N" anywhere in the subject + # is usually an issue reference, and treating it as a PR number would merge + # unrelated entries in merge_changelogs(). + match = re.search(r"\(#(\d+)\)$", subject) + number = int(match.group(1)) if match else None + if verbose: + label = f"#{number}" if number else sha[:12] + print(f"{label} - {subject} added.", file=sys.stderr) + + entry: dict = {"number": number, "title": subject} + if doc: + entry["sha"] = sha + entry["body"] = body.strip() + changelog.append(entry) + + return changelog + + +def merge_changelogs(milestone_entries: list[dict], git_entries: list[dict]) -> list[dict]: + """Union the milestone-sourced and git-sourced entries, deduplicated by PR number. + + Git commits keep their chronological order and carry commit-derived metadata. + Where a git commit's PR number matches a milestone PR, the milestone labels are + grafted onto the git entry. Milestone PRs with no corresponding commit in the + git range (e.g. folded into a combined backport) are appended, sorted by number. + """ + ms_by_number = {e["number"]: e for e in milestone_entries} + git_numbers = {e["number"] for e in git_entries if e.get("number")} + + for ge in git_entries: + num = ge.get("number") + if num in ms_by_number and ms_by_number[num].get("labels") and "labels" not in ge: + ge["labels"] = ms_by_number[num]["labels"] + + extras = [e for e in milestone_entries if e["number"] not in git_numbers] + extras.sort(key=lambda x: x["number"]) + + return git_entries + extras + + +def _lookup_milestone(client: httpx.Client, owner: str, repo: str, title: str) -> int | None: + page = 1 + while True: + resp = client.get(f"/repos/{owner}/{repo}/milestones", params={"state": "all", "per_page": 100, "page": page}) + _check_rate_limit(resp) + resp.raise_for_status() + data = resp.json() + if not data: + break + for ms in data: + if ms["title"] == title: + return ms["number"] + page += 1 + return None + + +def _http_status(response: str) -> int | None: + """Status code from the status line that `gh api --include` prints first.""" + match = re.match(r"HTTP/[\d.]+\s+(\d{3})", response.split("\n", 1)[0]) + return int(match.group(1)) if match else None + + +def _gh_is_merged(owner: str, repo: str, pr_number: int) -> bool: + """Whether pr_number is merged, via the gh CLI. + + gh exits non-zero for any HTTP error, so without the status line --include + supplies, a real 404 is indistinguishable from 403 rate limiting or a 5xx. + Anything but 204/404 is fatal; skipping would drop merged PRs and still exit 0. + """ + result = subprocess.run( + ["gh", "api", "--include", f"/repos/{owner}/{repo}/pulls/{pr_number}/merge"], + capture_output=True, + text=True, + ) + status = _http_status(result.stdout) + if status == 204: + return True + if status == 404: + return False + + print( + f"gh api error checking whether PR #{pr_number} is merged " + f"(HTTP {status if status else 'unknown'}): {result.stderr.strip()}", + file=sys.stderr, + ) + sys.exit(1) + + +def _is_merged(client: httpx.Client, owner: str, repo: str, pr_number: int) -> bool: + resp = client.get(f"/repos/{owner}/{repo}/pulls/{pr_number}/merge") + if resp.status_code == 204: + return True + if resp.status_code == 404: + return False + _check_rate_limit(resp) + resp.raise_for_status() + return False + + +def _check_rate_limit(resp: httpx.Response) -> None: + """Exit if @a resp is a rate-limit response, leaving other errors to the caller. + + A 403 alone does not identify a rate limit -- GitHub also uses it for a missing + token or insufficient scopes, where "try using an auth token" is wrong and hides + the real cause. A primary limit zeroes x-ratelimit-remaining, a secondary limit + sends retry-after, and a 429 is always a limit. Everything else falls through to + the raise_for_status() that follows each call site. + """ + if resp.status_code not in (403, 429): + return + if (resp.status_code == 429 or resp.headers.get("x-ratelimit-remaining") == "0" or "retry-after" in resp.headers): + print( + "You have exceeded your rate limit. Try using an auth token.", + file=sys.stderr, + ) + sys.exit(2) + + +def main() -> None: + parser = argparse.ArgumentParser(description="Generate changelog from merged PRs in a GitHub milestone.") + parser.add_argument("-o", "--owner", required=True, help="Repository owner") + parser.add_argument("-r", "--repo", required=True, help="Repository name") + parser.add_argument("-m", "--milestone", required=True, help="Milestone title") + parser.add_argument( + "-a", + "--auth", + default=None, + help="GitHub auth token. Discouraged: it is visible in ps output and shell history. Prefer GH_TOKEN.", + ) + parser.add_argument("-v", "--verbose", action="store_true", help="Verbose output") + parser.add_argument( + "--doc", + action="store_true", + help="Include extra metadata (merge SHA, PR body, labels) for documentation", + ) + parser.add_argument( + "--format", + choices=["text", "yaml"], + default="text", + help="Output format (default: text)", + ) + parser.add_argument( + "--use-gh", + action="store_true", + help="Use gh CLI instead of direct API calls (avoids rate limits)", + ) + parser.add_argument( + "--from-git", + metavar="RANGE", + default=None, + help="Also source commits from a git range (e.g. 10.1.3..upstream/10.1.x) and merge them " + "with the milestone PRs, deduplicated by PR number. Use for security releases whose fixes " + "land directly on a branch without public PRs. -m is still used as the version label.", + ) + args = parser.parse_args() + + token = args.auth or os.environ.get("GH_TOKEN") + + if not token and not args.use_gh: + print( + "WARNING: No GitHub token provided. Unauthenticated requests are limited\n" + "to 60 per hour, which is usually not enough to generate a full changelog.\n" + "\n" + "Provide a token via -a or the GH_TOKEN environment variable.\n" + "\n" + "To create a token:\n" + " 1. Go to https://github.com/settings/tokens\n"; + " 2. Click 'Generate new token' -> 'Generate new token (classic)'\n" + " 3. Select the 'public_repo' scope (sufficient for public repositories)\n" + " 4. For fine-grained tokens, grant read-only access to Issues and\n" + " Pull Requests on the target repository\n" + "\n" + "Alternatively, use --use-gh to use the gh CLI with its existing auth.\n", + file=sys.stderr, + ) + + if args.use_gh: + if not gh_cli_available(): + print("gh CLI not found. Install it or omit --use-gh.", file=sys.stderr) + sys.exit(1) + changelog = changelog_via_gh(args.owner, args.repo, args.milestone, args.verbose, args.doc) + else: + changelog = changelog_via_api( + args.owner, + args.repo, + args.milestone, + token, + args.verbose, + args.doc, + ) + + if args.from_git: + git_changelog = changelog_via_git(args.from_git, args.verbose, args.doc) + changelog = merge_changelogs(changelog, git_changelog) + + if changelog: + # Milestone-only entries all carry a PR number and sort by it. A merged + # (--from-git) result preserves git's chronological order with milestone-only + # PRs appended, so it must not be re-sorted. + if not args.from_git: + changelog.sort(key=lambda x: x["number"]) + + if args.format == "yaml": + output = { + "milestone": args.milestone, + "owner": args.owner, + "repo": args.repo, + "entries": changelog, + } + if yaml is None: + print("ERROR: --format yaml requires PyYAML. Install it with: pip install pyyaml", file=sys.stderr) + sys.exit(1) + yaml.dump(output, sys.stdout, default_flow_style=False, sort_keys=False, allow_unicode=True) Review Comment: The code treats PyYAML as an optional dependency (graceful ImportError + runtime error on `--format yaml`), but the project metadata currently declares `pyyaml>=6.0` as a required dependency. To keep behavior and packaging consistent, either (a) make PyYAML truly optional (declare it as an extra / optional dependency and keep the ImportError path), or (b) make it required in code (remove the try/except and the conditional runtime error since it should never trigger). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
