This is an automated email from the ASF dual-hosted git repository. tballison pushed a commit to branch add-site-update-skill in repository https://gitbox.apache.org/repos/asf/tika.git
commit bfc1beaa0f8480cf358ed1eada9c4e72cbf25c2a Author: tallison <[email protected]> AuthorDate: Tue Jul 21 16:07:53 2026 -0400 Add skill for updating the site --- .skills/update-site-for-release/SKILL.md | 329 +++++++++++++++++++++ .skills/update-site-for-release/scripts/README.md | 37 +++ .../scripts/extract-tika-contribs.py | 248 ++++++++++++++++ .../scripts/extract-tika-issues.py | 149 ++++++++++ .../scripts/scaffold-stable-version.sh | 65 ++++ 5 files changed, 828 insertions(+) diff --git a/.skills/update-site-for-release/SKILL.md b/.skills/update-site-for-release/SKILL.md new file mode 100644 index 0000000000..a11dfeb155 --- /dev/null +++ b/.skills/update-site-for-release/SKILL.md @@ -0,0 +1,329 @@ +--- +name: update-site-for-release +description: > + Update/publish the Apache Tika website (tika-site SVN repo) for a release — + step 17 of the Release Process. Handles STABLE (full per-version docs + + javadoc) vs PREVIEW/beta (Changes page + Antora docs). Use for "update the + site", "publish the site for X.Y.Z", "the website part of the release". +--- + +# Update the Tika website for a release + +Step 17 ("Update Tika site") of the Release Process +(<https://cwiki.apache.org/confluence/spaces/TIKA/pages/109454070/Release+Process>). +Assumes the release (tag, artifacts, VOTE, dist promotion) is done; covers the +**website** only. + +Set these first: +- **`$SITE`** — `tika-site` **SVN** checkout (not git): `src/site/` (sources) + + `publish/` (generated, SVN-tracked, served; `mvn install` regenerates it and + auto-runs `svn add --force publish` via antrun). +- **`$SCRATCH`** — release working dir: unzipped src release, `CHANGES-<NEW>.txt`, + built javadoc. + +Scripts: `./scripts/`. Local paths + toolchain (Maven binary, JDKs) live in a +private companion skill under `~/.claude/skills/`. + +> **[HUMAN] gates — never do these yourself:** final `svn commit` (outward-facing, +> irreversible), JIRA "release", s.apache.org shortlink, announce emails. Prepare +> everything, show `svn status`, hand off. + +--- + +## 0. Inputs + release type + +| Input | Example | Notes | +|---|---|---| +| `NEW_VERSION` | `3.3.2` | the release | +| `RELEASE_TYPE` | `stable` | `stable` or `preview` (alpha/beta/BETA) | +| `PREV_STABLE` / `PREV_PREVIEW` | `3.3.1` / `4.0.0-beta-1` | current `tika.stable.version` / `tika.preview.version` | +| `PREV_TAG` / `NEW_TAG` | `3.3.1` / `3.3.2` | git tags, for the GitHub contributor query | +| `JIRA fixVersion` | `3.3.2` | **may differ** from the label (betas often use the base version, e.g. `4.0.0`) | +| `CHANGES` file | `$SCRATCH/CHANGES-3.3.2.txt` | notable-changes source | +| src release zip | `tika-<NEW>-src.zip` | **stable only** (javadoc) | +| release date | `2026-07-21` | doap.rdf + news blurb | + +Confirm current values in `src/site/pom.xml`. + +| Step | STABLE | PREVIEW / beta | +|---|---|---| +| `pom.xml` `<parent><version>` | → `<NEW>` | leave at stable | +| `pom.xml` `tika.stable.version` | → `<NEW>` | leave | +| `pom.xml` `tika.preview.version` | leave | → `<NEW>` | +| `src/site/apt/<NEW>/` | full 8-file set (scaffold from prev) | **only `index.apt`** | +| `site.xml` entry | full sub-menu, expanded | minimal `Changes` item under `docs/<major>-SNAPSHOT` | +| formats.apt | regenerate from `tika-app` jar | n/a (Antora docs) | +| javadoc | `clean install -Pfast` + `javadoc:aggregate` → `publish/<NEW>/api` (step 7) | n/a | +| Antora docs | n/a | `mvn package -pl docs` + `docs/publish-docs.sh` | +| Download page | automatic | automatic | + +doap.rdf, index.apt.vm news, verify, publish are common to both. + +--- + +## 1. `src/site/pom.xml` versions [AGENT] + +- **STABLE:** bump `<parent><version>` **and** `<tika.stable.version>` to `<NEW>`. + (parent version drives `${project.parent.version}` home-page links → must be + newest stable.) +- **PREVIEW:** bump only `<tika.preview.version>`; leave parent + stable. + +Download page auto-reads these — no manual edit. + +--- + +## 2. `src/site/site.xml` menu [AGENT] + +Current stable + current preview expanded; older = `collapse="true"`. + +- **STABLE:** new expanded block above the previous stable; add `collapse="true"` + to the old stable block: + ```xml + <item name="Apache Tika 3.3.2" href="3.3.2/index.html"> + <item name="Getting Started" href="3.3.2/gettingstarted.html"/> + <item name="Supported Formats" href="3.3.2/formats.html"/> + <item name="Parser API" href="3.3.2/parser.html"/> + <item name="Parser 5min Quick Start Guide" href="3.3.2/parser_guide.html"/> + <item name="Content and Language Detection" href="3.3.2/detection.html"/> + <item name="Configuring Tika" href="3.3.2/configuring.html"/> + <item name="Usage Examples" href="3.3.2/examples.html"/> + <item name="API Documentation" href="3.3.2/api/"/> + </item> + ``` +- **PREVIEW:** minimal block above the old preview (collapse it); `href` → Antora + snapshot, not an apt page: + ```xml + <item name="Apache Tika 4.0.0-beta-2" href="docs/4.0.0-SNAPSHOT"> + <item name="Changes" href="4.0.0-beta-2/index.html"/> + </item> + ``` + +--- + +## 3. Per-version apt docs `src/site/apt/<NEW>/` [AGENT] + +**STABLE** — scaffold (these docs are version-string-identical across 3.x): +```bash +./scripts/scaffold-stable-version.sh $SITE 3.3.1 3.3.2 +``` +Copies+bumps `configuring/detection/examples/parser/parser_guide/gettingstarted.apt`. Then: +- **formats.apt** — two parts: a hand-written top (license, intro, `%{toc}`, ~25 + prose format-family sections) down to the header line + `Full list of Supported Formats in "standard" artifacts`, then a generated flat + list below it. `--list-parser-details-apt` regenerates only the flat part. Copy + prev `formats.apt`, bump versions, replace everything **below** that header: + ```bash + java -jar <path>/tika-app-<NEW>.jar --list-parser-details-apt + ``` + **Do NOT truncate at the `%{toc}` line** — keep the prose sections. (Scaffold + skips `formats.apt` for this reason.) +- **index.apt** — step 4. + +**PREVIEW** — create only `index.apt` (copy prev preview's; title `Apache Tika +<NEW>`; fill step 4). No other files. + +--- + +## 4. Per-version `index.apt`: notable changes + contributors [AGENT + HUMAN] + +Shape (see `src/site/apt/3.3.1/index.apt`): license+title; "most notable changes…" +bullets; "The following people have contributed…" bullets; "See +{{https://s.apache.org/XXXX}} …". + +**Notable changes** — review output, keep only notable items: +```bash +./scripts/extract-tika-issues.py CHANGES-3.3.2.txt out-3.3.2.apt 3.3.2 +``` +Mirrors CHANGES verbatim; TIKA-####/Github-#### auto-linked; ALL-CAPS headers → +apt sections. + +**Contributors** — candidate list, RM curates: +```bash +./scripts/extract-tika-contribs.py 3.3.2 --prev-tag 3.3.1 --tag 3.3.2 > contribs.txt +# beta (fixVersion differs from label): +# ./scripts/extract-tika-contribs.py 4.0.0 --prev-tag 3.3.1 --tag 4.0.0-beta-1 +``` +Merges JIRA (reporters/assignees/comment authors) + GitHub commit/PR authors +(uses `gh` auth; resolves logins→names; case-insensitive sort; filters bots/AI). +Over-reports drive-by commenters, misses GitHub-issue-only commenters. **[HUMAN]** +prune / normalise / add. + +**Shortlink [HUMAN]** — `s.apache.org/XXXX` → the JIRA "issues fixed in <NEW>" +query; needs s.apache.org login. Ask the RM. + +--- + +## 5. `src/site/resources/doap.rdf` [AGENT] + +New `<release>` at the top. Ordering is by **date, not version** (a stable point +release can sit above an older-dated preview — 3.3.2/Jul-16 above 4.0.0-beta-1/Jul-3): +```xml + <release> + <Version> + <name>Apache Tika 3.3.2</name> + <created>2026-07-21</created> + <revision>3.3.2</revision> + </Version> + </release> +``` + +--- + +## 6. Home page `src/site/apt/index.apt.vm` [AGENT + HUMAN] + +1. New **Latest News** block at the top; its CHANGES link uses + `dist.apache.org/repos/dist/release/...` (live mirror): + ``` + [21 July 2026: Apache Tika Release] + Apache Tika 3.3.2 has been released! <one or two sentence summary>. + Please see the {{{https://dist.apache.org/repos/dist/release/tika/3.3.2/CHANGES-3.3.2.txt}CHANGES.txt}} + file for the full list of changes in the release and have a look at the download page for more information + on how to obtain Apache Tika 3.3.2. + ``` +2. **Repoint the superseded release's CHANGES link** (its artifacts get `svn rm`'d + from the live mirror at release): `dist.apache.org/repos/dist/release/tika/<PREV>/…` + → `archive.apache.org/dist/tika/<PREV>/…`. **[HUMAN]** confirm which version was + removed. + +--- + +## 7. Docs / Javadoc + +**STABLE — javadoc [AGENT].** NOT the wiki's `javadoc:aggregate-no-fork` (runs +against `tika-parent`, its relative `<sourcepath>` fails → `No source files for +package org.apache.tika`; wrong goal, not a JDK issue). From the unzipped src +release (its `./mvnw` is broken — use system `mvn`): +```bash +unzip tika-3.3.2-src.zip && cd tika-3.3.2 +mvn clean install -Pfast # ~4 min; module artifacts + full dep classpath +mvn javadoc:aggregate # FORKING goal (NOT -no-fork) +mkdir -p $SITE/publish/3.3.2 +mv target/reports/apidocs $SITE/publish/3.3.2/api +``` +Both steps matter: without `install` javadoc dies on `package org.slf4j does not +exist`; the forking `aggregate` (@aggregator) runs once on the root, `-no-fork` +breaks per-`pom`-module. Any modern JDK (11 and 25 verified). (`tika-server` +miredot docs discontinued — skip.) + +**PREVIEW (4.x) — Antora docs [AGENT]:** +```bash +cd tika-<NEW> # unzipped src release +./mvnw package -pl docs +./docs/publish-docs.sh $SITE/publish +``` +`publish-docs.sh` copies target/site into `publish/docs/`, flattens URLs, rewrites +the search index (has its own guards). + +--- + +## 8. Build + verify [AGENT] + +`tika-site` has no `./mvnw` → system `mvn`. **Build with Java 11** — it pins +maven-site-plugin 3.4 (2014), unreliable on newer JDKs; a Doxia error here means +wrong JDK, not a content problem (separate from step 7's JDK-agnostic javadoc). +```bash +cd "$SITE" +mvn clean install +``` +> **ALWAYS `clean install`, never bare `install`** — an incremental build leaves +> `publish/css/` stale → pages render with no CSS/sidebar. Fix is a `clean` +> rebuild, not a CSS edit. + +Build auto-copies target/site → `publish/`, strips timestamps, `svn add --force +publish`. Check: new version in the menu; news + download versions right; **pages +styled (CSS + sidebar)**; per-version pages + javadoc/Antora resolve. Preview: +`mvn site:run` → <http://localhost:8080>. + +--- + +## 9. Stage + hand off the commit [HUMAN] + +```bash +cd "$SITE" +svn status +svn add src/site/apt/<NEW> # + any other new files +# hand to the RM — do NOT run yourself: +# svn commit -m "Update website for <NEW> release." +``` + +**Big-commit caveat (stable):** `publish/<NEW>/api` is ~3,000 files / ~55 MB; a +single commit often **times out / `E000104 Connection reset by peer`** — this is +size, NOT auth (bad password = `Authentication failed`/403, and cached creds won't +re-prompt). Fixes: +- `http-timeout = 1800` in `~/.subversion/servers` `[global]`. +- Else commit the api in chunks, then the rest: + ```bash + svn commit --depth=empty publish/<NEW> publish/<NEW>/api \ + publish/<NEW>/api/org publish/<NEW>/api/org/apache \ + publish/<NEW>/api/org/apache/tika -m "<NEW> site: api dir skeleton" + for d in publish/<NEW>/api/org/apache/tika/*/; do + svn commit "$d" -m "<NEW> javadoc: $(basename "$d")" || break # parser/ ~1,300 files + done + svn commit publish/<NEW>/api -m "<NEW> javadoc: remaining api files" + svn commit -m "Update website for <NEW> release." + ``` +- Atomic per invocation → a failed commit rolls back; retry. Locked (`E155004`) → + `svn cleanup`. + +--- + +## 10. Confirm published + re-kick [HUMAN] + +svnwcsub maps `/www/tika.apache.org ← %(ASF)s/tika/site/publish`: **only a commit +touching `publish/` triggers a republish**, and it publishes the whole tree at +HEAD. Verify (cache-buster hits the origin, not Varnish): +```bash +curl -s -o /dev/null -w "%{http_code}\n" "https://tika.apache.org/<NEW>/index.html?cb=$(date +%s)" +``` +Want `200`. Still `404`/old minutes later → the web-node `svn up` choked on the +big commit. **Re-kick** with a trivial whitespace commit to a file **under +`publish/`** (e.g. a blank line in `publish/index.html`): +```bash +svn commit publish/index.html -m "Nudge svnwcsub to republish." +``` +Re-fires svnwcsub → `svn up` to HEAD (harmless; next build regenerates it). A +commit outside `publish/` won't trigger. Still stuck ~30 min → ping `#asfinfra`. + +--- + +## 11. Post-site [HUMAN] (context) + +- JIRA: "release" `<NEW>`; move stragglers to the next version. +- Announce to `user@`, `dev@`, `[email protected]`. +- Log at <https://reporter.apache.org/addrelease.html?tika>. +- CVEs fixed → update `security.apt` / per-version security notes, republish. + +--- + +## Checklist + +- [ ] stable vs preview decided +- [ ] `pom.xml` versions (parent+stable for stable; preview only for preview) +- [ ] `site.xml`: new entry added, previous same-track entry collapsed +- [ ] per-version apt docs (full set for stable / `index.apt` only for preview) +- [ ] `formats.apt` regenerated (stable) +- [ ] `index.apt`: notable changes + curated contributors + shortlink +- [ ] `doap.rdf` entry +- [ ] `index.apt.vm`: news block + superseded CHANGES link → archive +- [ ] javadoc → `publish/<NEW>/api` (stable) / Antora docs (preview) +- [ ] `mvn clean install` (never bare `install`); pages styled (CSS + sidebar) +- [ ] `svn status`/`svn add` done, commit handed to RM (chunk `api/` if it resets) +- [ ] live site 200: `https://tika.apache.org/<NEW>/index.html?cb=…` — else re-kick + +--- + +## Troubleshooting + +| Symptom | Cause | Fix | +|---|---|---| +| `aggregate-no-fork` → `No source files for package org.apache.tika` | runs against `tika-parent`; relative `<sourcepath>` can't resolve | use forking `javadoc:aggregate` after `clean install -Pfast` (step 7) | +| javadoc → `package org.slf4j does not exist` etc. | aggregate without a prior build → empty classpath | `mvn clean install -Pfast` first | +| pages unstyled (no CSS/sidebar) | incremental `install` left `publish/css/` stale | `mvn clean install` (never bare `install`) | +| site-plugin / Doxia error on `mvn install` | maven-site-plugin 3.4 on too-new a JDK | build with **Java 11** | +| notable-changes bullet split on a version number | old numeric heuristic (removed) | use the bundled script; re-run | +| contributors have bots/AI, or surname order | old behavior (fixed): filter + `str.casefold` sort | use bundled `extract-tika-contribs.py`; RM curates | +| commit `E175012 timed out` / `E000104 Connection reset` | ~55 MB api tree too big for one transaction | `http-timeout=1800`; chunk the `api/` (step 9). Size, not auth. | +| commit `Authentication failed` / 403 | genuinely bad/expired credential | `svn commit --username <you>` to re-cache | +| `svn: E155004 working copy locked` | prior commit died mid-transaction | `svn cleanup`, retry | +| committed, site still old even with `?cb=` (origin 404s/old) | web-node `svn up` choked on the big commit | re-kick: whitespace commit under `publish/` (step 10); stuck ~30 min → `#asfinfra` | +| home-page CHANGES link 404s for the previous release | it was `svn rm`'d from the live dist mirror | repoint to `archive.apache.org/dist/tika/<prev>/CHANGES-<prev>.txt` (step 6) | diff --git a/.skills/update-site-for-release/scripts/README.md b/.skills/update-site-for-release/scripts/README.md new file mode 100644 index 0000000000..e55ab0a387 --- /dev/null +++ b/.skills/update-site-for-release/scripts/README.md @@ -0,0 +1,37 @@ +# Helper scripts for `update-site-for-release` + +Python 3. No third-party deps (stdlib only). Run from anywhere. + +| Script | Purpose | Key gotcha | +|---|---|---| +| `extract-tika-issues.py` | `CHANGES-X.Y.Z.txt` → apt "notable changes" bullet list | Mirrors CHANGES verbatim (bullets + all-caps section headers); review to keep only notable items. Arg order is `CHANGES OUTPUT VERSION` (guarded against the common swap). | +| `extract-tika-contribs.py` | Candidate contributor list from **JIRA + GitHub** | **Candidate only** — RM must curate. `FIX_VERSION` may differ from the release label (betas). Pass `--prev-tag`/`--tag` to include GitHub PR authors. Uses `gh auth token` automatically. | +| `scaffold-stable-version.sh` | Copy previous stable's apt docs → new version, bump version string | STABLE only. Skips `formats.apt`/`index.apt` on purpose (regenerate those). | + +## Origin + +`extract-tika-issues.py` and the idea behind `extract-tika-contribs.py` come from +Chris Mattmann's [`chrismattmann/apachestuff`](https://github.com/chrismattmann/apachestuff) +repo. The original `extract-tika-contribs` was a bash one-liner that scraped the +JIRA "printable" search HTML (`grep ViewProfile | cut | sort | uniq`) behind a +`tika` shell alias — fragile, capped at 100 rows, and blind to GitHub-only +contributors. This version queries the JIRA REST API and the GitHub compare API +instead. (`apachestuff` also has `name_to_committer_id.py`; the Tika site process +lists contributors by display name and never uses committer ids, so it is +intentionally not bundled here.) + +## Examples + +```bash +# Notable changes for a stable release +./extract-tika-issues.py /path/to/CHANGES-3.3.2.txt out-3.3.2.apt 3.3.2 + +# Contributors, stable (JIRA fixVersion == label) +./extract-tika-contribs.py 3.3.2 --prev-tag 3.3.1 --tag 3.3.2 + +# Contributors, beta (JIRA fixVersion is the base version, tag is the label) +./extract-tika-contribs.py 4.0.0 --prev-tag 3.3.1 --tag 4.0.0-beta-1 + +# Scaffold the new stable version's apt docs +./scaffold-stable-version.sh /path/to/tika-site 3.3.1 3.3.2 +``` diff --git a/.skills/update-site-for-release/scripts/extract-tika-contribs.py b/.skills/update-site-for-release/scripts/extract-tika-contribs.py new file mode 100755 index 0000000000..d6d35d3466 --- /dev/null +++ b/.skills/update-site-for-release/scripts/extract-tika-contribs.py @@ -0,0 +1,248 @@ +#!/usr/bin/env python3 +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +""" +Emit a CANDIDATE Apache-Tika contributor list (apt bullet format) for a release, +merging two sources: + + 1. JIRA -- reporters, assignees, and comment authors on every issue in the + fix version. (Replaces Chris Mattmann's original `extract-tika-contribs` + bash script, chrismattmann/apachestuff, which scraped the JIRA printable + HTML via `grep ViewProfile | cut | sort | uniq` behind a `tika` shell alias.) + + 2. GitHub -- commit authors and Co-authored-by trailers between the previous + release tag and the new one. Increasingly contributors go straight to + GitHub and never touch JIRA, so this catches PR authors the JIRA query + alone misses. + + STILL A CANDIDATE, not the final list. It over-reports (drive-by commenters, + bots) and can under-report -- e.g. someone whose only contribution was a + comment on a GitHub *issue* (not a PR) appears in neither source. The tika-site + 3.3.1 index.apt lists "Chengxin Xu", who is absent from both JIRA fixVersion + 3.3.1 and the 3.3.0..3.3.1 git range. The release manager MUST review the + output, prune noise, normalise names, and add anyone missing. + +Usage: + extract-tika-contribs.py FIX_VERSION [--prev-tag T] [--tag T] + [--repo apache/tika] [--project TIKA] + [--no-github] [--include-bots] + + FIX_VERSION JIRA fixVersion. NOT always the release label -- for a + beta, issues are often tagged with the base version + (4.0.0-beta-1 contributors came from fixVersion "4.0.0"). + --prev-tag / --tag git tags bounding this release, e.g. --prev-tag 3.3.1 + --tag 3.3.2. If omitted, the GitHub half is skipped. + --no-github JIRA only. + +GitHub auth: uses GITHUB_TOKEN / GH_TOKEN if set, else `gh auth token`, else +anonymous (60 req/hr). `gh` is the easiest -- if you can run gh, this just works. + +Examples: + ./extract-tika-contribs.py 3.3.2 --prev-tag 3.3.1 --tag 3.3.2 + ./extract-tika-contribs.py 4.0.0 --prev-tag 3.3.1 --tag 4.0.0-beta-1 +""" +import argparse +import json +import re +import subprocess +import sys +import urllib.parse +import urllib.request + +JIRA = "https://issues.apache.org/jira"; +GITHUB = "https://api.github.com"; + +# Automated / non-human accounts (JIRA display names or GitHub author names). +# Matched case-insensitively as substrings. +BOT_DENYLIST = [ + "asf github bot", "asf subversion and git services", "hudson", "jenkins", + "gridgain integration", "jira bot", "githubbot", + "dependabot", "[bot]", "copilot", "github-actions", + # AI coding assistants that show up as commit/co-authors. Kept specific so a + # real person named e.g. "Claude Warren" is NOT filtered. + "claude sonnet", "claude opus", "claude haiku", "claude code", + "chatgpt", "openai", "gpt-4", "gpt-5", "cursor agent", "devin ai", +] + +# Co-author email domains that identify AI assistants regardless of display name. +AI_EMAIL_DOMAINS = ["anthropic.com", "openai.com"] + + +def is_bot(name): + low = name.lower() + return any(bot in low for bot in BOT_DENYLIST) + + +def _get_json(url, headers): + req = urllib.request.Request(url, headers=headers) + with urllib.request.urlopen(req, timeout=60) as resp: + return json.load(resp) + + +# ---------------------------------------------------------------- JIRA ---- +def collect_jira(fix_version, project): + jql = f'project = {project} AND fixVersion = "{fix_version}"' + names, start_at, total, issue_count = set(), 0, None, 0 + while total is None or start_at < total: + params = urllib.parse.urlencode({ + "jql": jql, "fields": "reporter,assignee,comment", + "startAt": start_at, "maxResults": 100, + }) + data = _get_json(f"{JIRA}/rest/api/2/search?{params}", {"Accept": "application/json"}) + total = data.get("total", 0) + issues = data.get("issues", []) + if not issues: + break + for issue in issues: + issue_count += 1 + f = issue.get("fields", {}) + for who in (f.get("reporter"), f.get("assignee")): + if who and who.get("displayName"): + names.add(who["displayName"].strip()) + for c in (f.get("comment") or {}).get("comments", []): + a = c.get("author") or {} + if a.get("displayName"): + names.add(a["displayName"].strip()) + start_at += len(issues) + return names, issue_count, total or 0 + + +# -------------------------------------------------------------- GitHub ---- +def _github_token(): + import os + tok = os.environ.get("GITHUB_TOKEN") or os.environ.get("GH_TOKEN") + if tok: + return tok.strip() + try: + out = subprocess.run(["gh", "auth", "token"], capture_output=True, text=True, timeout=15) + if out.returncode == 0 and out.stdout.strip(): + return out.stdout.strip() + except (FileNotFoundError, subprocess.SubprocessError): + pass + return None + + +def _resolve_login(login, headers, cache): + """GitHub login -> display name (falls back to the login). Collapses + login-vs-real-name duplicates like THausherr / Tilman Hausherr.""" + if login in cache: + return cache[login] + name = login + try: + data = _get_json(f"{GITHUB}/users/{login}", headers) + if data.get("name"): + name = data["name"].strip() + except Exception: # noqa: BLE001 - keep the login on any lookup failure + pass + cache[login] = name + return name + + +def collect_github(repo, prev_tag, tag): + headers = {"Accept": "application/vnd.github+json", "User-Agent": "update-site-for-release"} + tok = _github_token() + if tok: + headers["Authorization"] = f"Bearer {tok}" + names, logins, page, total_commits = set(), set(), 1, None + while True: + url = f"{GITHUB}/repos/{repo}/compare/{prev_tag}...{tag}?per_page=100&page={page}" + data = _get_json(url, headers) + if "commits" not in data: + raise RuntimeError(data.get("message", "unexpected GitHub response")) + total_commits = data.get("total_commits", total_commits) + commits = data["commits"] + if not commits: + break + for c in commits: + login = (c.get("author") or {}).get("login") + if login: + logins.add(login) + else: + # No linked GitHub account (rare) -- fall back to the git author name. + a = c["commit"]["author"] + if a.get("name"): + names.add(a["name"].strip()) + for m in re.finditer(r"Co-authored-by:\s*(.+?)\s*<([^>]*)>", c["commit"]["message"]): + nm, email = m.group(1).strip(), m.group(2).strip().lower() + if any(dom in email for dom in AI_EMAIL_DOMAINS): + continue + names.add(nm) + if len(commits) < 100: + break + page += 1 + # Resolve logins to display names so they dedupe against JIRA/co-author names. + cache = {} + for login in logins: + names.add(_resolve_login(login, headers, cache)) + return names, total_commits or 0, bool(tok) + + +# ---------------------------------------------------------------- main ---- +def main(): + ap = argparse.ArgumentParser(description="Candidate Tika contributor list (JIRA + GitHub).") + ap.add_argument("fix_version") + ap.add_argument("--prev-tag") + ap.add_argument("--tag") + ap.add_argument("--repo", default="apache/tika") + ap.add_argument("--project", default="TIKA") + ap.add_argument("--no-github", action="store_true") + ap.add_argument("--include-bots", action="store_true") + args = ap.parse_args() + + all_names = set() + notes = [] + + try: + jira_names, issue_count, total = collect_jira(args.fix_version, args.project) + all_names |= jira_names + notes.append(f"JIRA fixVersion={args.fix_version}: {len(jira_names)} name(s) " + f"from {issue_count} issue(s).") + except Exception as e: # noqa: BLE001 + notes.append(f"JIRA query FAILED: {e}") + + do_github = not args.no_github and args.prev_tag and args.tag + if do_github: + try: + gh_names, ncommits, authed = collect_github(args.repo, args.prev_tag, args.tag) + all_names |= gh_names + notes.append(f"GitHub {args.prev_tag}..{args.tag}: {len(gh_names)} " + f"name(s) from {ncommits} commit(s) " + f"[{'authenticated' if authed else 'anonymous'}].") + except Exception as e: # noqa: BLE001 + notes.append(f"GitHub query FAILED: {e}") + elif not args.no_github: + notes.append("GitHub half SKIPPED: pass --prev-tag and --tag to include PR authors.") + + if not args.include_bots: + all_names = {n for n in all_names if not is_bot(n)} + + # Simple case-insensitive alphabetical sort by the full display string + # (first-name-first, as written) -- the site's historical convention, which + # came from the old bash script piping through `sort`. NOT a last-name sort. + for name in sorted(all_names, key=str.casefold): + print(f" * {name}") + + print("", file=sys.stderr) + for n in notes: + print(f"# {n}", file=sys.stderr) + print(f"# {len(all_names)} candidate contributor(s) total. CANDIDATE LIST -- review " + f"before use: prune drive-by/bot names, normalise duplicates (login vs real " + f"name), and add GitHub-issue commenters not captured by either source.", + file=sys.stderr) + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/.skills/update-site-for-release/scripts/extract-tika-issues.py b/.skills/update-site-for-release/scripts/extract-tika-issues.py new file mode 100755 index 0000000000..c4b73c6ca6 --- /dev/null +++ b/.skills/update-site-for-release/scripts/extract-tika-issues.py @@ -0,0 +1,149 @@ +#!/usr/bin/env python3 +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +""" +Turn a CHANGES-X.Y.Z.txt release section into the apt "notable changes" list. + +Python 3 port of Chris Mattmann's extract-tika-issues.py (chrismattmann/apachestuff), +extended to recognise " BREAKING CHANGES" / " NEW FEATURES" style section headers +and to join wrapped continuation lines. + +Usage: + extract-tika-issues.py CHANGES_FILE OUTPUT_FILE RELEASE_VERSION + + CHANGES_FILE path to CHANGES-X.Y.Z.txt (input) + OUTPUT_FILE path to write the apt fragment (output; will be overwritten) + RELEASE_VERSION e.g. 3.3.2 -- must match a "Release X.Y.Z - MM/dd/yyyy" header + +Example: + ./extract-tika-issues.py CHANGES-3.3.2.txt out-3.3.2.apt 3.3.2 + +NOTE: the output is a STARTING POINT, not final copy. The numeric heuristic +mis-splits sentences that contain version numbers (e.g. "Jetty 11 -> 12.0.36" +becomes a bogus bullet). Hand-edit the result before pasting it into index.apt, +and keep only the genuinely notable items -- this is not meant to be the whole +changelog. +""" +import re +import sys + +USAGE = ("Usage: extract-tika-issues.py CHANGES_FILE OUTPUT_FILE RELEASE_VERSION\n" + " e.g. extract-tika-issues.py CHANGES-3.3.2.txt out-3.3.2.apt 3.3.2") + + +def looks_like_version(s): + return re.fullmatch(r"\d+\.\d+\.\d+(?:[-.].+)?", s.strip()) is not None + + +def main(): + if len(sys.argv) != 4: + print(USAGE, file=sys.stderr) + return 2 + + changesFile, outputFile, relVersion = sys.argv[1], sys.argv[2], sys.argv[3] + + # Guard the classic arg-order slip (OUTPUT and VERSION swapped), which + # silently writes a file literally named after the output path. + if looks_like_version(outputFile) and not looks_like_version(relVersion): + print(f"ERROR: argument 2 ('{outputFile}') looks like a version and argument 3 " + f"('{relVersion}') does not.\n" + f"Arguments are: CHANGES_FILE OUTPUT_FILE RELEASE_VERSION -- did you swap " + f"the last two?\n{USAGE}", file=sys.stderr) + return 2 + + versionChangeLog = {} + version = None + + # Tika CHANGES.txt grammar: + # "Release X.Y.Z - M/D/YYYY" version header + # " * text ..." a changelog entry (may wrap over indented lines) + # " BREAKING CHANGES" an all-caps section header (optional; some releases) + # blank line separates entries + release_re = re.compile(r".*\d{1,2}/\d{1,2}/\d{4}.*") + bullet_re = re.compile(r"^\s*\*\s+(.*)") + # All-caps line = section header. Restricting to A-Z and spaces (no digits, + # slashes, or lowercase) keeps wrapped prose from being mistaken for a header. + # NOTE: the old numeric-list heuristic was removed -- it mangled entries + # containing version numbers ("Port the 4.x SAX-based parsers..." -> "x SAX..."). + section_re = re.compile(r"^[A-Z][A-Z ]{2,38}$") + SECTION_MARKER = "__SECTION__:" + + with open(changesFile, mode="r", encoding="utf-8") as cf: + lines = cf.readlines() + + versionIssues = [] + issueTxt = "" + for line in lines: + stripped = line.strip() + + if release_re.match(line.rstrip()): + if issueTxt: + versionIssues.append(issueTxt) + issueTxt = "" + if version is not None: + versionChangeLog[version] = versionIssues + versionIssues = [] + version = line.rsplit(" - ", 1)[0].strip() + continue + if version is None or not stripped: # preamble or entry separator + continue + if "notable changes" in line or set(stripped) <= {"-"}: + continue + m = bullet_re.match(line) + if m: # new changelog entry + if issueTxt: + versionIssues.append(issueTxt) + issueTxt = m.group(1).strip() + elif section_re.match(stripped): # section header + if issueTxt: + versionIssues.append(issueTxt) + issueTxt = "" + versionIssues.append(SECTION_MARKER + stripped.title()) + else: # continuation of current entry + issueTxt = (issueTxt + " " + stripped) if issueTxt else stripped + + if issueTxt: + versionIssues.append(issueTxt) + if version is not None: + versionChangeLog[version] = versionIssues + + key = "Release " + relVersion + if key not in versionChangeLog: + print(f"ERROR: no '{key}' section found in {changesFile}.\n" + f"Sections present: {sorted(versionChangeLog)}", file=sys.stderr) + return 1 + + with open(outputFile, "w", encoding="utf-8") as of: + of.write("Apache Tika " + relVersion + "\n\n") + of.write("\t The most notable changes in Tika " + relVersion + + " over the previous release are:\n\n") + for issue in versionChangeLog[key]: + if issue.startswith(SECTION_MARKER): + of.write("* " + issue[len(SECTION_MARKER):] + "\n\n") + continue + out = re.sub(r"TIKA-(\d+)", + r"{{{http://issues.apache.org/jira/browse/TIKA-\1}TIKA-\1}}";, issue) + out = re.sub(r"Github-(\d+)", + r"{{{http://github.com/apache/tika/pull/\1}Github-\1}}";, out, flags=re.I) + of.write("\t * " + out + "\n\n") + + print(f"Wrote {outputFile} for release {relVersion} " + f"({len(versionChangeLog[key])} entries). Review & hand-edit before pasting.", + file=sys.stderr) + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/.skills/update-site-for-release/scripts/scaffold-stable-version.sh b/.skills/update-site-for-release/scripts/scaffold-stable-version.sh new file mode 100644 index 0000000000..fa8709f619 --- /dev/null +++ b/.skills/update-site-for-release/scripts/scaffold-stable-version.sh @@ -0,0 +1,65 @@ +#!/bin/bash +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +# Scaffold a new STABLE version's apt docs by copying the previous stable +# version's directory and bumping the version string. Safe because the +# per-version apt docs (configuring/detection/examples/parser/parser_guide/ +# gettingstarted) are identical across 3.x releases apart from the version +# string. formats.apt and index.apt are intentionally NOT copied: they must be +# regenerated (formats from `tika-app --list-parser-details-apt`, index from +# the extract-tika-* scripts + hand-editing). +# +# PREVIEW/beta releases do NOT use this -- they get only a hand-written +# index.apt. See SKILL.md. +# +# Usage: +# scaffold-stable-version.sh <tika-site-root> <prev-version> <new-version> +# Example: +# scaffold-stable-version.sh /path/to/tika-site 3.3.1 3.3.2 +set -euo pipefail + +SITE_ROOT="${1:?usage: scaffold-stable-version.sh <tika-site-root> <prev-version> <new-version>}" +PREV="${2:?missing <prev-version>}" +NEW="${3:?missing <new-version>}" + +APT="${SITE_ROOT}/src/site/apt" +SRC="${APT}/${PREV}" +DST="${APT}/${NEW}" + +[[ -d "${SRC}" ]] || { echo "Previous version dir not found: ${SRC}" >&2; exit 1; } +[[ -e "${DST}" ]] && { echo "Target already exists: ${DST} (refusing to overwrite)" >&2; exit 1; } + +# Files that are pure version-string bumps of the previous release. +COPY_BUMP=(configuring.apt detection.apt examples.apt parser.apt parser_guide.apt gettingstarted.apt) + +mkdir -p "${DST}" +for f in "${COPY_BUMP[@]}"; do + if [[ -f "${SRC}/${f}" ]]; then + sed "s/${PREV//./\\.}/${NEW}/g" "${SRC}/${f}" > "${DST}/${f}" + echo " bumped ${f}" + else + echo " WARN: ${SRC}/${f} missing, skipped" >&2 + fi +done + +echo "" +echo "Scaffolded ${DST} from ${PREV}." +echo "STILL TO DO (not done by this script):" +echo " * formats.apt -- regenerate: java -jar tika-app-${NEW}.jar --list-parser-details-apt" +echo " (prepend the same header block as ${SRC}/formats.apt, then the list)" +echo " * index.apt -- build from extract-tika-issues.py + extract-tika-contribs.py, hand-edit" +echo " * git/svn add -- 'svn add ${DST}' once the two files above exist" +echo " * Verify each bumped file's version references look right (diff against ${SRC})."
