This is an automated email from the ASF dual-hosted git repository.
tballison pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/tika.git
The following commit(s) were added to refs/heads/main by this push:
new 69b45d5d5d Add site update skill (#2963)
69b45d5d5d is described below
commit 69b45d5d5d12437969f42a4e19131c4f7d3bb1ef
Author: Tim Allison <[email protected]>
AuthorDate: Wed Jul 22 09:34:10 2026 -0400
Add site update skill (#2963)
---
.skills/update-site-for-release/SKILL.md | 346 +++++++++++++++++++++
.skills/update-site-for-release/scripts/README.md | 54 ++++
.../scripts/extract-tika-contribs.py | 252 +++++++++++++++
.../scripts/extract-tika-issues.py | 152 +++++++++
.../scripts/scaffold-stable-version.sh | 65 ++++
5 files changed, 869 insertions(+)
diff --git a/.skills/update-site-for-release/SKILL.md
b/.skills/update-site-for-release/SKILL.md
new file mode 100644
index 0000000000..784d58ae60
--- /dev/null
+++ b/.skills/update-site-for-release/SKILL.md
@@ -0,0 +1,346 @@
+---
+name: update-site-for-release
+description: >
+ Update/publish the Apache Tika website (tika-site SVN repo) for a release —
+ step 17 of the Release Process. Handles STABLE (full per-version docs +
+ javadoc) vs PREVIEW/beta (Changes page + Antora docs). Use for "update the
+ site", "publish the site for X.Y.Z", "the website part of the release".
+---
+
+<!--
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to You under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# Update the Tika website for a release
+
+Step 17 ("Update Tika site") of the Release Process
+(<https://cwiki.apache.org/confluence/spaces/TIKA/pages/109454070/Release+Process>).
+Assumes the release (tag, artifacts, VOTE, dist promotion) is done; covers the
+**website** only.
+
+Set these first:
+- **`$SITE`** — `tika-site` **SVN** checkout (not git): `src/site/` (sources) +
+ `publish/` (generated, SVN-tracked, served; `mvn install` regenerates it and
+ auto-runs `svn add --force publish` via antrun).
+- **`$SCRATCH`** — release working dir: unzipped src release,
`CHANGES-<NEW>.txt`,
+ built javadoc.
+
+Scripts: `./scripts/`. Local paths + toolchain (Maven binary, JDKs) live in a
+private companion skill under `~/.claude/skills/`.
+
+> **[HUMAN] gates — never do these yourself:** final `svn commit`
(outward-facing,
+> irreversible), JIRA "release", s.apache.org shortlink, announce emails.
Prepare
+> everything, show `svn status`, hand off.
+
+---
+
+## 0. Inputs + release type
+
+| Input | Example | Notes |
+|---|---|---|
+| `NEW_VERSION` | `3.3.2` | the release |
+| `RELEASE_TYPE` | `stable` | `stable` or `preview` (alpha/beta/BETA) |
+| `PREV_STABLE` / `PREV_PREVIEW` | `3.3.1` / `4.0.0-beta-1` | current
`tika.stable.version` / `tika.preview.version` |
+| `PREV_TAG` / `NEW_TAG` | `3.3.1` / `3.3.2` | git tags, for the GitHub
contributor query |
+| `JIRA fixVersion` | `3.3.2` | **may differ** from the label (betas often use
the base version, e.g. `4.0.0`) |
+| `CHANGES` file | `$SCRATCH/CHANGES-3.3.2.txt` | notable-changes source |
+| src release zip | `tika-<NEW>-src.zip` | **stable only** (javadoc) |
+| release date | `2026-07-21` | doap.rdf + news blurb |
+
+Confirm current values in `src/site/pom.xml`.
+
+| Step | STABLE | PREVIEW / beta |
+|---|---|---|
+| `pom.xml` `<parent><version>` | → `<NEW>` | leave at stable |
+| `pom.xml` `tika.stable.version` | → `<NEW>` | leave |
+| `pom.xml` `tika.preview.version` | leave | → `<NEW>` |
+| `src/site/apt/<NEW>/` | full 8-file set (scaffold from prev) | **only
`index.apt`** |
+| `site.xml` entry | full sub-menu, expanded | minimal `Changes` item under
`docs/<major>-SNAPSHOT` |
+| formats.apt | regenerate from `tika-app` jar | n/a (Antora docs) |
+| javadoc | `clean install -Pfast` + `javadoc:aggregate` → `publish/<NEW>/api`
(step 7) | n/a |
+| Antora docs | n/a | `mvn package -pl docs` + `docs/publish-docs.sh` |
+| Download page | automatic | automatic |
+
+doap.rdf, index.apt.vm news, verify, publish are common to both.
+
+---
+
+## 1. `src/site/pom.xml` versions [AGENT]
+
+- **STABLE:** bump `<parent><version>` **and** `<tika.stable.version>` to
`<NEW>`.
+ (parent version drives `${project.parent.version}` home-page links → must be
+ newest stable.)
+- **PREVIEW:** bump only `<tika.preview.version>`; leave parent + stable.
+
+Download page auto-reads these — no manual edit.
+
+---
+
+## 2. `src/site/site.xml` menu [AGENT]
+
+Current stable + current preview expanded; older = `collapse="true"`.
+
+- **STABLE:** new expanded block above the previous stable; add
`collapse="true"`
+ to the old stable block:
+ ```xml
+ <item name="Apache Tika 3.3.2" href="3.3.2/index.html">
+ <item name="Getting Started"
href="3.3.2/gettingstarted.html"/>
+ <item name="Supported Formats" href="3.3.2/formats.html"/>
+ <item name="Parser API" href="3.3.2/parser.html"/>
+ <item name="Parser 5min Quick Start Guide"
href="3.3.2/parser_guide.html"/>
+ <item name="Content and Language Detection" href="3.3.2/detection.html"/>
+ <item name="Configuring Tika" href="3.3.2/configuring.html"/>
+ <item name="Usage Examples" href="3.3.2/examples.html"/>
+ <item name="API Documentation" href="3.3.2/api/"/>
+ </item>
+ ```
+- **PREVIEW:** minimal block above the old preview (collapse it); `href` →
Antora
+ snapshot, not an apt page:
+ ```xml
+ <item name="Apache Tika 4.0.0-beta-2" href="docs/4.0.0-SNAPSHOT">
+ <item name="Changes" href="4.0.0-beta-2/index.html"/>
+ </item>
+ ```
+
+---
+
+## 3. Per-version apt docs `src/site/apt/<NEW>/` [AGENT]
+
+**STABLE** — scaffold (these docs are version-string-identical across 3.x):
+```bash
+./scripts/scaffold-stable-version.sh $SITE 3.3.1 3.3.2
+```
+Copies+bumps
`configuring/detection/examples/parser/parser_guide/gettingstarted.apt`. Then:
+- **formats.apt** — two parts: a hand-written top (license, intro, `%{toc}`,
~25
+ prose format-family sections) down to the header line
+ `Full list of Supported Formats in "standard" artifacts`, then a generated
flat
+ list below it. `--list-parser-details-apt` regenerates only the flat part.
Copy
+ prev `formats.apt`, bump versions, replace everything **below** that header:
+ ```bash
+ java -jar <path>/tika-app-<NEW>.jar --list-parser-details-apt
+ ```
+ **Do NOT truncate at the `%{toc}` line** — keep the prose sections. (Scaffold
+ skips `formats.apt` for this reason.)
+- **index.apt** — step 4.
+
+**PREVIEW** — create only `index.apt` (copy prev preview's; title `Apache Tika
+<NEW>`; fill step 4). No other files.
+
+---
+
+## 4. Per-version `index.apt`: notable changes + contributors [AGENT + HUMAN]
+
+Shape (see `src/site/apt/3.3.1/index.apt`): license+title; "most notable
changes…"
+bullets; "The following people have contributed…" bullets; "See
+{{https://s.apache.org/XXXX}} …".
+
+**Notable changes** — review output, keep only notable items:
+```bash
+./scripts/extract-tika-issues.py CHANGES-3.3.2.txt out-3.3.2.apt 3.3.2
+```
+Mirrors CHANGES verbatim; TIKA-####/Github-#### auto-linked; ALL-CAPS headers →
+apt sections.
+
+**Contributors** — candidate list, RM curates:
+```bash
+./scripts/extract-tika-contribs.py 3.3.2 --prev-tag 3.3.1 --tag 3.3.2 >
contribs.txt
+# beta (fixVersion differs from label):
+# ./scripts/extract-tika-contribs.py 4.0.0 --prev-tag 3.3.1 --tag 4.0.0-beta-1
+```
+Merges JIRA (reporters/assignees/comment authors) + GitHub commit/PR authors
+(uses `gh` auth; resolves logins→names; case-insensitive sort; filters
bots/AI).
+Over-reports drive-by commenters, misses GitHub-issue-only commenters.
**[HUMAN]**
+prune / normalise / add.
+
+**Shortlink [HUMAN]** — `s.apache.org/XXXX` → the JIRA "issues fixed in <NEW>"
+query; needs s.apache.org login. Ask the RM.
+
+---
+
+## 5. `src/site/resources/doap.rdf` [AGENT]
+
+New `<release>` at the top. Ordering is by **date, not version** (a stable
point
+release can sit above an older-dated preview — 3.3.2/Jul-16 above
4.0.0-beta-1/Jul-3):
+```xml
+ <release>
+ <Version>
+ <name>Apache Tika 3.3.2</name>
+ <created>2026-07-21</created>
+ <revision>3.3.2</revision>
+ </Version>
+ </release>
+```
+
+---
+
+## 6. Home page `src/site/apt/index.apt.vm` [AGENT + HUMAN]
+
+1. New **Latest News** block at the top; its CHANGES link uses
+ `dist.apache.org/repos/dist/release/...` (live mirror):
+ ```
+ [21 July 2026: Apache Tika Release]
+ Apache Tika 3.3.2 has been released! <one or two sentence summary>.
+ Please see the
{{{https://dist.apache.org/repos/dist/release/tika/3.3.2/CHANGES-3.3.2.txt}CHANGES.txt}}
+ file for the full list of changes in the release and have a look at the
download page for more information
+ on how to obtain Apache Tika 3.3.2.
+ ```
+2. **Repoint the superseded release's CHANGES link** (its artifacts get `svn
rm`'d
+ from the live mirror at release):
`dist.apache.org/repos/dist/release/tika/<PREV>/…`
+ → `archive.apache.org/dist/tika/<PREV>/…`. **[HUMAN]** confirm which
version was
+ removed.
+
+---
+
+## 7. Docs / Javadoc
+
+**STABLE — javadoc [AGENT].** NOT the wiki's `javadoc:aggregate-no-fork` (runs
+against `tika-parent`, its relative `<sourcepath>` fails → `No source files for
+package org.apache.tika`; wrong goal, not a JDK issue). From the unzipped src
+release (its `./mvnw` is broken — use system `mvn`):
+```bash
+unzip tika-3.3.2-src.zip && cd tika-3.3.2
+mvn clean install -Pfast # ~4 min; module artifacts + full dep classpath
+mvn javadoc:aggregate # FORKING goal (NOT -no-fork)
+mkdir -p $SITE/publish/3.3.2
+mv target/reports/apidocs $SITE/publish/3.3.2/api
+```
+Both steps matter: without `install` javadoc dies on `package org.slf4j does
not
+exist`; the forking `aggregate` (@aggregator) runs once on the root, `-no-fork`
+breaks per-`pom`-module. Any modern JDK (11 and 25 verified). (`tika-server`
+miredot docs discontinued — skip.)
+
+**PREVIEW (4.x) — Antora docs [AGENT]:**
+```bash
+cd tika-<NEW> # unzipped src release
+./mvnw package -pl docs
+./docs/publish-docs.sh $SITE/publish
+```
+`publish-docs.sh` copies target/site into `publish/docs/`, flattens URLs,
rewrites
+the search index (has its own guards).
+
+---
+
+## 8. Build + verify [AGENT]
+
+`tika-site` has no `./mvnw` → system `mvn`. **Build with Java 11** — it pins
+maven-site-plugin 3.4 (2014), unreliable on newer JDKs; a Doxia error here
means
+wrong JDK, not a content problem (separate from step 7's JDK-agnostic javadoc).
+```bash
+cd "$SITE"
+mvn clean install
+```
+> **ALWAYS `clean install`, never bare `install`** — an incremental build
leaves
+> `publish/css/` stale → pages render with no CSS/sidebar. Fix is a `clean`
+> rebuild, not a CSS edit.
+
+Build auto-copies target/site → `publish/`, strips timestamps, `svn add --force
+publish`. Check: new version in the menu; news + download versions right;
**pages
+styled (CSS + sidebar)**; per-version pages + javadoc/Antora resolve. Preview:
+`mvn site:run` → <http://localhost:8080>.
+
+---
+
+## 9. Stage + hand off the commit [HUMAN]
+
+```bash
+cd "$SITE"
+svn status
+svn add src/site/apt/<NEW> # + any other new files
+# hand to the RM — do NOT run yourself:
+# svn commit -m "Update website for <NEW> release."
+```
+
+**Big-commit caveat (stable):** `publish/<NEW>/api` is ~3,000 files / ~55 MB; a
+single commit often **times out / `E000104 Connection reset by peer`** — this
is
+size, NOT auth (bad password = `Authentication failed`/403, and cached creds
won't
+re-prompt). Fixes:
+- `http-timeout = 1800` in `~/.subversion/servers` `[global]`.
+- Else commit the api in chunks, then the rest:
+ ```bash
+ svn commit --depth=empty publish/<NEW> publish/<NEW>/api \
+ publish/<NEW>/api/org publish/<NEW>/api/org/apache \
+ publish/<NEW>/api/org/apache/tika -m "<NEW> site: api dir skeleton"
+ for d in publish/<NEW>/api/org/apache/tika/*/; do
+ svn commit "$d" -m "<NEW> javadoc: $(basename "$d")" || break # parser/
~1,300 files
+ done
+ svn commit publish/<NEW>/api -m "<NEW> javadoc: remaining api files"
+ svn commit -m "Update website for <NEW> release."
+ ```
+- Atomic per invocation → a failed commit rolls back; retry. Locked
(`E155004`) →
+ `svn cleanup`.
+
+---
+
+## 10. Confirm published + re-kick [HUMAN]
+
+svnwcsub maps `/www/tika.apache.org ← %(ASF)s/tika/site/publish`: **only a
commit
+touching `publish/` triggers a republish**, and it publishes the whole tree at
+HEAD. Verify (cache-buster hits the origin, not Varnish):
+```bash
+curl -s -o /dev/null -w "%{http_code}\n"
"https://tika.apache.org/<NEW>/index.html?cb=$(date +%s)"
+```
+Want `200`. Still `404`/old minutes later → the web-node `svn up` choked on the
+big commit. **Re-kick** with a trivial whitespace commit to a file **under
+`publish/`** (e.g. a blank line in `publish/index.html`):
+```bash
+svn commit publish/index.html -m "Nudge svnwcsub to republish."
+```
+Re-fires svnwcsub → `svn up` to HEAD (harmless; next build regenerates it). A
+commit outside `publish/` won't trigger. Still stuck ~30 min → ping
`#asfinfra`.
+
+---
+
+## 11. Post-site [HUMAN] (context)
+
+- JIRA: "release" `<NEW>`; move stragglers to the next version.
+- Announce to `user@`, `dev@`, `[email protected]`.
+- Log at <https://reporter.apache.org/addrelease.html?tika>.
+- CVEs fixed → update `security.apt` / per-version security notes, republish.
+
+---
+
+## Checklist
+
+- [ ] stable vs preview decided
+- [ ] `pom.xml` versions (parent+stable for stable; preview only for preview)
+- [ ] `site.xml`: new entry added, previous same-track entry collapsed
+- [ ] per-version apt docs (full set for stable / `index.apt` only for preview)
+- [ ] `formats.apt` regenerated (stable)
+- [ ] `index.apt`: notable changes + curated contributors + shortlink
+- [ ] `doap.rdf` entry
+- [ ] `index.apt.vm`: news block + superseded CHANGES link → archive
+- [ ] javadoc → `publish/<NEW>/api` (stable) / Antora docs (preview)
+- [ ] `mvn clean install` (never bare `install`); pages styled (CSS + sidebar)
+- [ ] `svn status`/`svn add` done, commit handed to RM (chunk `api/` if it
resets)
+- [ ] live site 200: `https://tika.apache.org/<NEW>/index.html?cb=…` — else
re-kick
+
+---
+
+## Troubleshooting
+
+| Symptom | Cause | Fix |
+|---|---|---|
+| `aggregate-no-fork` → `No source files for package org.apache.tika` | runs
against `tika-parent`; relative `<sourcepath>` can't resolve | use forking
`javadoc:aggregate` after `clean install -Pfast` (step 7) |
+| javadoc → `package org.slf4j does not exist` etc. | aggregate without a
prior build → empty classpath | `mvn clean install -Pfast` first |
+| pages unstyled (no CSS/sidebar) | incremental `install` left `publish/css/`
stale | `mvn clean install` (never bare `install`) |
+| site-plugin / Doxia error on `mvn install` | maven-site-plugin 3.4 on
too-new a JDK | build with **Java 11** |
+| notable-changes bullet split on a version number | old numeric heuristic
(removed) | use the bundled script; re-run |
+| contributors have bots/AI, or surname order | old behavior (fixed): filter +
`str.casefold` sort | use bundled `extract-tika-contribs.py`; RM curates |
+| commit `E175012 timed out` / `E000104 Connection reset` | ~55 MB api tree
too big for one transaction | `http-timeout=1800`; chunk the `api/` (step 9).
Size, not auth. |
+| commit `Authentication failed` / 403 | genuinely bad/expired credential |
`svn commit --username <you>` to re-cache |
+| `svn: E155004 working copy locked` | prior commit died mid-transaction |
`svn cleanup`, retry |
+| committed, site still old even with `?cb=` (origin 404s/old) | web-node `svn
up` choked on the big commit | re-kick: whitespace commit under `publish/`
(step 10); stuck ~30 min → `#asfinfra` |
+| home-page CHANGES link 404s for the previous release | it was `svn rm`'d
from the live dist mirror | repoint to
`archive.apache.org/dist/tika/<prev>/CHANGES-<prev>.txt` (step 6) |
diff --git a/.skills/update-site-for-release/scripts/README.md
b/.skills/update-site-for-release/scripts/README.md
new file mode 100644
index 0000000000..be38e8d080
--- /dev/null
+++ b/.skills/update-site-for-release/scripts/README.md
@@ -0,0 +1,54 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to You under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# Helper scripts for `update-site-for-release`
+
+Python 3. No third-party deps (stdlib only). Run from anywhere.
+
+| Script | Purpose | Key gotcha |
+|---|---|---|
+| `extract-tika-issues.py` | `CHANGES-X.Y.Z.txt` → apt "notable changes"
bullet list | Mirrors CHANGES verbatim (bullets + all-caps section headers);
review to keep only notable items. Arg order is `CHANGES OUTPUT VERSION`
(guarded against the common swap). |
+| `extract-tika-contribs.py` | Candidate contributor list from **JIRA +
GitHub** | **Candidate only** — RM must curate. `FIX_VERSION` may differ from
the release label (betas). Pass `--prev-tag`/`--tag` to include GitHub PR
authors. Uses `gh auth token` automatically. |
+| `scaffold-stable-version.sh` | Copy previous stable's apt docs → new
version, bump version string | STABLE only. Skips `formats.apt`/`index.apt` on
purpose (regenerate those). |
+
+## Origin
+
+`extract-tika-issues.py` and the idea behind `extract-tika-contribs.py` come
from
+Chris Mattmann's
[`chrismattmann/apachestuff`](https://github.com/chrismattmann/apachestuff)
+repo. The original `extract-tika-contribs` was a bash one-liner that scraped
the
+JIRA "printable" search HTML (`grep ViewProfile | cut | sort | uniq`) behind a
+`tika` shell alias — fragile, capped at 100 rows, and blind to GitHub-only
+contributors. This version queries the JIRA REST API and the GitHub compare API
+instead. (`apachestuff` also has `name_to_committer_id.py`; the Tika site
process
+lists contributors by display name and never uses committer ids, so it is
+intentionally not bundled here.)
+
+## Examples
+
+```bash
+# Notable changes for a stable release
+./extract-tika-issues.py /path/to/CHANGES-3.3.2.txt out-3.3.2.apt 3.3.2
+
+# Contributors, stable (JIRA fixVersion == label)
+./extract-tika-contribs.py 3.3.2 --prev-tag 3.3.1 --tag 3.3.2
+
+# Contributors, beta (JIRA fixVersion is the base version, tag is the label)
+./extract-tika-contribs.py 4.0.0 --prev-tag 3.3.1 --tag 4.0.0-beta-1
+
+# Scaffold the new stable version's apt docs
+./scaffold-stable-version.sh /path/to/tika-site 3.3.1 3.3.2
+```
diff --git a/.skills/update-site-for-release/scripts/extract-tika-contribs.py
b/.skills/update-site-for-release/scripts/extract-tika-contribs.py
new file mode 100755
index 0000000000..ec9387cae2
--- /dev/null
+++ b/.skills/update-site-for-release/scripts/extract-tika-contribs.py
@@ -0,0 +1,252 @@
+#!/usr/bin/env python3
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""
+Emit a CANDIDATE Apache-Tika contributor list (apt bullet format) for a
release,
+merging two sources:
+
+ 1. JIRA -- reporters, assignees, and comment authors on every issue in the
+ fix version. (Reimplements Chris Mattmann's original
`extract-tika-contribs`
+ bash script, from
+
https://github.com/chrismattmann/apachestuff/blob/master/extract-tika-contribs
+ which scraped the JIRA printable HTML via `grep ViewProfile | cut | sort |
+ uniq` behind a `tika` shell alias. This version uses the JIRA REST +
GitHub
+ compare APIs instead and adds GitHub PR authors, login->name resolution,
+ bot/AI filtering, and a case-insensitive sort.)
+
+ 2. GitHub -- commit authors and Co-authored-by trailers between the previous
+ release tag and the new one. Increasingly contributors go straight to
+ GitHub and never touch JIRA, so this catches PR authors the JIRA query
+ alone misses.
+
+ STILL A CANDIDATE, not the final list. It over-reports (drive-by commenters,
+ bots) and can under-report -- e.g. someone whose only contribution was a
+ comment on a GitHub *issue* (not a PR) appears in neither source. The
tika-site
+ 3.3.1 index.apt lists "Chengxin Xu", who is absent from both JIRA fixVersion
+ 3.3.1 and the 3.3.0..3.3.1 git range. The release manager MUST review the
+ output, prune noise, normalise names, and add anyone missing.
+
+Usage:
+ extract-tika-contribs.py FIX_VERSION [--prev-tag T] [--tag T]
+ [--repo apache/tika] [--project TIKA]
+ [--no-github] [--include-bots]
+
+ FIX_VERSION JIRA fixVersion. NOT always the release label -- for a
+ beta, issues are often tagged with the base version
+ (4.0.0-beta-1 contributors came from fixVersion
"4.0.0").
+ --prev-tag / --tag git tags bounding this release, e.g. --prev-tag 3.3.1
+ --tag 3.3.2. If omitted, the GitHub half is skipped.
+ --no-github JIRA only.
+
+GitHub auth: uses GITHUB_TOKEN / GH_TOKEN if set, else `gh auth token`, else
+anonymous (60 req/hr). `gh` is the easiest -- if you can run gh, this just
works.
+
+Examples:
+ ./extract-tika-contribs.py 3.3.2 --prev-tag 3.3.1 --tag 3.3.2
+ ./extract-tika-contribs.py 4.0.0 --prev-tag 3.3.1 --tag 4.0.0-beta-1
+"""
+import argparse
+import json
+import re
+import subprocess
+import sys
+import urllib.parse
+import urllib.request
+
+JIRA = "https://issues.apache.org/jira";
+GITHUB = "https://api.github.com";
+
+# Automated / non-human accounts (JIRA display names or GitHub author names).
+# Matched case-insensitively as substrings.
+BOT_DENYLIST = [
+ "asf github bot", "asf subversion and git services", "hudson", "jenkins",
+ "gridgain integration", "jira bot", "githubbot",
+ "dependabot", "[bot]", "copilot", "github-actions",
+ # AI coding assistants that show up as commit/co-authors. Kept specific so
a
+ # real person named e.g. "Claude Warren" is NOT filtered.
+ "claude sonnet", "claude opus", "claude haiku", "claude code",
+ "chatgpt", "openai", "gpt-4", "gpt-5", "cursor agent", "devin ai",
+]
+
+# Co-author email domains that identify AI assistants regardless of display
name.
+AI_EMAIL_DOMAINS = ["anthropic.com", "openai.com"]
+
+
+def is_bot(name):
+ low = name.lower()
+ return any(bot in low for bot in BOT_DENYLIST)
+
+
+def _get_json(url, headers):
+ req = urllib.request.Request(url, headers=headers)
+ with urllib.request.urlopen(req, timeout=60) as resp:
+ return json.load(resp)
+
+
+# ---------------------------------------------------------------- JIRA ----
+def collect_jira(fix_version, project):
+ jql = f'project = {project} AND fixVersion = "{fix_version}"'
+ names, start_at, total, issue_count = set(), 0, None, 0
+ while total is None or start_at < total:
+ params = urllib.parse.urlencode({
+ "jql": jql, "fields": "reporter,assignee,comment",
+ "startAt": start_at, "maxResults": 100,
+ })
+ data = _get_json(f"{JIRA}/rest/api/2/search?{params}", {"Accept":
"application/json"})
+ total = data.get("total", 0)
+ issues = data.get("issues", [])
+ if not issues:
+ break
+ for issue in issues:
+ issue_count += 1
+ f = issue.get("fields", {})
+ for who in (f.get("reporter"), f.get("assignee")):
+ if who and who.get("displayName"):
+ names.add(who["displayName"].strip())
+ for c in (f.get("comment") or {}).get("comments", []):
+ a = c.get("author") or {}
+ if a.get("displayName"):
+ names.add(a["displayName"].strip())
+ start_at += len(issues)
+ return names, issue_count, total or 0
+
+
+# -------------------------------------------------------------- GitHub ----
+def _github_token():
+ import os
+ tok = os.environ.get("GITHUB_TOKEN") or os.environ.get("GH_TOKEN")
+ if tok:
+ return tok.strip()
+ try:
+ out = subprocess.run(["gh", "auth", "token"], capture_output=True,
text=True, timeout=15)
+ if out.returncode == 0 and out.stdout.strip():
+ return out.stdout.strip()
+ except (FileNotFoundError, subprocess.SubprocessError):
+ pass
+ return None
+
+
+def _resolve_login(login, headers, cache):
+ """GitHub login -> display name (falls back to the login). Collapses
+ login-vs-real-name duplicates like THausherr / Tilman Hausherr."""
+ if login in cache:
+ return cache[login]
+ name = login
+ try:
+ data = _get_json(f"{GITHUB}/users/{login}", headers)
+ if data.get("name"):
+ name = data["name"].strip()
+ except Exception: # noqa: BLE001 - keep the login on any lookup failure
+ pass
+ cache[login] = name
+ return name
+
+
+def collect_github(repo, prev_tag, tag):
+ headers = {"Accept": "application/vnd.github+json", "User-Agent":
"update-site-for-release"}
+ tok = _github_token()
+ if tok:
+ headers["Authorization"] = f"Bearer {tok}"
+ names, logins, page, total_commits = set(), set(), 1, None
+ while True:
+ url =
f"{GITHUB}/repos/{repo}/compare/{prev_tag}...{tag}?per_page=100&page={page}"
+ data = _get_json(url, headers)
+ if "commits" not in data:
+ raise RuntimeError(data.get("message", "unexpected GitHub
response"))
+ total_commits = data.get("total_commits", total_commits)
+ commits = data["commits"]
+ if not commits:
+ break
+ for c in commits:
+ login = (c.get("author") or {}).get("login")
+ if login:
+ logins.add(login)
+ else:
+ # No linked GitHub account (rare) -- fall back to the git
author name.
+ a = c["commit"]["author"]
+ if a.get("name"):
+ names.add(a["name"].strip())
+ for m in re.finditer(r"Co-authored-by:\s*(.+?)\s*<([^>]*)>",
c["commit"]["message"]):
+ nm, email = m.group(1).strip(), m.group(2).strip().lower()
+ if any(dom in email for dom in AI_EMAIL_DOMAINS):
+ continue
+ names.add(nm)
+ if len(commits) < 100:
+ break
+ page += 1
+ # Resolve logins to display names so they dedupe against JIRA/co-author
names.
+ cache = {}
+ for login in logins:
+ names.add(_resolve_login(login, headers, cache))
+ return names, total_commits or 0, bool(tok)
+
+
+# ---------------------------------------------------------------- main ----
+def main():
+ ap = argparse.ArgumentParser(description="Candidate Tika contributor list
(JIRA + GitHub).")
+ ap.add_argument("fix_version")
+ ap.add_argument("--prev-tag")
+ ap.add_argument("--tag")
+ ap.add_argument("--repo", default="apache/tika")
+ ap.add_argument("--project", default="TIKA")
+ ap.add_argument("--no-github", action="store_true")
+ ap.add_argument("--include-bots", action="store_true")
+ args = ap.parse_args()
+
+ all_names = set()
+ notes = []
+
+ try:
+ jira_names, issue_count, total = collect_jira(args.fix_version,
args.project)
+ all_names |= jira_names
+ notes.append(f"JIRA fixVersion={args.fix_version}: {len(jira_names)}
name(s) "
+ f"from {issue_count} issue(s).")
+ except Exception as e: # noqa: BLE001
+ notes.append(f"JIRA query FAILED: {e}")
+
+ do_github = not args.no_github and args.prev_tag and args.tag
+ if do_github:
+ try:
+ gh_names, ncommits, authed = collect_github(args.repo,
args.prev_tag, args.tag)
+ all_names |= gh_names
+ notes.append(f"GitHub {args.prev_tag}..{args.tag}: {len(gh_names)}
"
+ f"name(s) from {ncommits} commit(s) "
+ f"[{'authenticated' if authed else 'anonymous'}].")
+ except Exception as e: # noqa: BLE001
+ notes.append(f"GitHub query FAILED: {e}")
+ elif not args.no_github:
+ notes.append("GitHub half SKIPPED: pass --prev-tag and --tag to
include PR authors.")
+
+ if not args.include_bots:
+ all_names = {n for n in all_names if not is_bot(n)}
+
+ # Simple case-insensitive alphabetical sort by the full display string
+ # (first-name-first, as written) -- the site's historical convention, which
+ # came from the old bash script piping through `sort`. NOT a last-name
sort.
+ for name in sorted(all_names, key=str.casefold):
+ print(f" * {name}")
+
+ print("", file=sys.stderr)
+ for n in notes:
+ print(f"# {n}", file=sys.stderr)
+ print(f"# {len(all_names)} candidate contributor(s) total. CANDIDATE LIST
-- review "
+ f"before use: prune drive-by/bot names, normalise duplicates (login
vs real "
+ f"name), and add GitHub-issue commenters not captured by either
source.",
+ file=sys.stderr)
+ return 0
+
+
+if __name__ == "__main__":
+ sys.exit(main())
diff --git a/.skills/update-site-for-release/scripts/extract-tika-issues.py
b/.skills/update-site-for-release/scripts/extract-tika-issues.py
new file mode 100755
index 0000000000..99cb40d968
--- /dev/null
+++ b/.skills/update-site-for-release/scripts/extract-tika-issues.py
@@ -0,0 +1,152 @@
+#!/usr/bin/env python3
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""
+Turn a CHANGES-X.Y.Z.txt release section into the apt "notable changes" list.
+
+Python 3 port of Chris Mattmann's extract-tika-issues.py, originally from:
+
https://github.com/chrismattmann/apachestuff/blob/master/extract-tika-issues.py
+Updates over the original: py2 -> py3; dropped the numeric-list heuristic that
+mangled version numbers ("4.x", "12.0.36"); recognise " BREAKING CHANGES" /
+" NEW FEATURES" section headers; join wrapped continuation lines; guard the
+common arg-order swap.
+
+Usage:
+ extract-tika-issues.py CHANGES_FILE OUTPUT_FILE RELEASE_VERSION
+
+ CHANGES_FILE path to CHANGES-X.Y.Z.txt (input)
+ OUTPUT_FILE path to write the apt fragment (output; will be overwritten)
+ RELEASE_VERSION e.g. 3.3.2 -- must match a "Release X.Y.Z - MM/dd/yyyy"
header
+
+Example:
+ ./extract-tika-issues.py CHANGES-3.3.2.txt out-3.3.2.apt 3.3.2
+
+NOTE: the output is a STARTING POINT, not final copy. The numeric heuristic
+mis-splits sentences that contain version numbers (e.g. "Jetty 11 -> 12.0.36"
+becomes a bogus bullet). Hand-edit the result before pasting it into index.apt,
+and keep only the genuinely notable items -- this is not meant to be the whole
+changelog.
+"""
+import re
+import sys
+
+USAGE = ("Usage: extract-tika-issues.py CHANGES_FILE OUTPUT_FILE
RELEASE_VERSION\n"
+ " e.g. extract-tika-issues.py CHANGES-3.3.2.txt out-3.3.2.apt 3.3.2")
+
+
+def looks_like_version(s):
+ return re.fullmatch(r"\d+\.\d+\.\d+(?:[-.].+)?", s.strip()) is not None
+
+
+def main():
+ if len(sys.argv) != 4:
+ print(USAGE, file=sys.stderr)
+ return 2
+
+ changesFile, outputFile, relVersion = sys.argv[1], sys.argv[2], sys.argv[3]
+
+ # Guard the classic arg-order slip (OUTPUT and VERSION swapped), which
+ # silently writes a file literally named after the output path.
+ if looks_like_version(outputFile) and not looks_like_version(relVersion):
+ print(f"ERROR: argument 2 ('{outputFile}') looks like a version and
argument 3 "
+ f"('{relVersion}') does not.\n"
+ f"Arguments are: CHANGES_FILE OUTPUT_FILE RELEASE_VERSION -- did
you swap "
+ f"the last two?\n{USAGE}", file=sys.stderr)
+ return 2
+
+ versionChangeLog = {}
+ version = None
+
+ # Tika CHANGES.txt grammar:
+ # "Release X.Y.Z - M/D/YYYY" version header
+ # " * text ..." a changelog entry (may wrap over indented
lines)
+ # " BREAKING CHANGES" an all-caps section header (optional; some
releases)
+ # blank line separates entries
+ release_re = re.compile(r".*\d{1,2}/\d{1,2}/\d{4}.*")
+ bullet_re = re.compile(r"^\s*\*\s+(.*)")
+ # All-caps line = section header. Restricting to A-Z and spaces (no digits,
+ # slashes, or lowercase) keeps wrapped prose from being mistaken for a
header.
+ # NOTE: the old numeric-list heuristic was removed -- it mangled entries
+ # containing version numbers ("Port the 4.x SAX-based parsers..." -> "x
SAX...").
+ section_re = re.compile(r"^[A-Z][A-Z ]{2,38}$")
+ SECTION_MARKER = "__SECTION__:"
+
+ with open(changesFile, mode="r", encoding="utf-8") as cf:
+ lines = cf.readlines()
+
+ versionIssues = []
+ issueTxt = ""
+ for line in lines:
+ stripped = line.strip()
+
+ if release_re.match(line.rstrip()):
+ if issueTxt:
+ versionIssues.append(issueTxt)
+ issueTxt = ""
+ if version is not None:
+ versionChangeLog[version] = versionIssues
+ versionIssues = []
+ version = line.rsplit(" - ", 1)[0].strip()
+ continue
+ if version is None or not stripped: # preamble or entry
separator
+ continue
+ if "notable changes" in line or set(stripped) <= {"-"}:
+ continue
+ m = bullet_re.match(line)
+ if m: # new changelog entry
+ if issueTxt:
+ versionIssues.append(issueTxt)
+ issueTxt = m.group(1).strip()
+ elif section_re.match(stripped): # section header
+ if issueTxt:
+ versionIssues.append(issueTxt)
+ issueTxt = ""
+ versionIssues.append(SECTION_MARKER + stripped.title())
+ else: # continuation of
current entry
+ issueTxt = (issueTxt + " " + stripped) if issueTxt else stripped
+
+ if issueTxt:
+ versionIssues.append(issueTxt)
+ if version is not None:
+ versionChangeLog[version] = versionIssues
+
+ key = "Release " + relVersion
+ if key not in versionChangeLog:
+ print(f"ERROR: no '{key}' section found in {changesFile}.\n"
+ f"Sections present: {sorted(versionChangeLog)}", file=sys.stderr)
+ return 1
+
+ with open(outputFile, "w", encoding="utf-8") as of:
+ of.write("Apache Tika " + relVersion + "\n\n")
+ of.write("\t The most notable changes in Tika " + relVersion +
+ " over the previous release are:\n\n")
+ for issue in versionChangeLog[key]:
+ if issue.startswith(SECTION_MARKER):
+ of.write("* " + issue[len(SECTION_MARKER):] + "\n\n")
+ continue
+ out = re.sub(r"TIKA-(\d+)",
+
r"{{{http://issues.apache.org/jira/browse/TIKA-\1}TIKA-\1}}";, issue)
+ out = re.sub(r"Github-(\d+)",
+
r"{{{http://github.com/apache/tika/pull/\1}Github-\1}}";, out, flags=re.I)
+ of.write("\t * " + out + "\n\n")
+
+ print(f"Wrote {outputFile} for release {relVersion} "
+ f"({len(versionChangeLog[key])} entries). Review & hand-edit before
pasting.",
+ file=sys.stderr)
+ return 0
+
+
+if __name__ == "__main__":
+ sys.exit(main())
diff --git a/.skills/update-site-for-release/scripts/scaffold-stable-version.sh
b/.skills/update-site-for-release/scripts/scaffold-stable-version.sh
new file mode 100644
index 0000000000..fa8709f619
--- /dev/null
+++ b/.skills/update-site-for-release/scripts/scaffold-stable-version.sh
@@ -0,0 +1,65 @@
+#!/bin/bash
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# Scaffold a new STABLE version's apt docs by copying the previous stable
+# version's directory and bumping the version string. Safe because the
+# per-version apt docs (configuring/detection/examples/parser/parser_guide/
+# gettingstarted) are identical across 3.x releases apart from the version
+# string. formats.apt and index.apt are intentionally NOT copied: they must be
+# regenerated (formats from `tika-app --list-parser-details-apt`, index from
+# the extract-tika-* scripts + hand-editing).
+#
+# PREVIEW/beta releases do NOT use this -- they get only a hand-written
+# index.apt. See SKILL.md.
+#
+# Usage:
+# scaffold-stable-version.sh <tika-site-root> <prev-version> <new-version>
+# Example:
+# scaffold-stable-version.sh /path/to/tika-site 3.3.1 3.3.2
+set -euo pipefail
+
+SITE_ROOT="${1:?usage: scaffold-stable-version.sh <tika-site-root>
<prev-version> <new-version>}"
+PREV="${2:?missing <prev-version>}"
+NEW="${3:?missing <new-version>}"
+
+APT="${SITE_ROOT}/src/site/apt"
+SRC="${APT}/${PREV}"
+DST="${APT}/${NEW}"
+
+[[ -d "${SRC}" ]] || { echo "Previous version dir not found: ${SRC}" >&2; exit
1; }
+[[ -e "${DST}" ]] && { echo "Target already exists: ${DST} (refusing to
overwrite)" >&2; exit 1; }
+
+# Files that are pure version-string bumps of the previous release.
+COPY_BUMP=(configuring.apt detection.apt examples.apt parser.apt
parser_guide.apt gettingstarted.apt)
+
+mkdir -p "${DST}"
+for f in "${COPY_BUMP[@]}"; do
+ if [[ -f "${SRC}/${f}" ]]; then
+ sed "s/${PREV//./\\.}/${NEW}/g" "${SRC}/${f}" > "${DST}/${f}"
+ echo " bumped ${f}"
+ else
+ echo " WARN: ${SRC}/${f} missing, skipped" >&2
+ fi
+done
+
+echo ""
+echo "Scaffolded ${DST} from ${PREV}."
+echo "STILL TO DO (not done by this script):"
+echo " * formats.apt -- regenerate: java -jar tika-app-${NEW}.jar
--list-parser-details-apt"
+echo " (prepend the same header block as ${SRC}/formats.apt,
then the list)"
+echo " * index.apt -- build from extract-tika-issues.py +
extract-tika-contribs.py, hand-edit"
+echo " * git/svn add -- 'svn add ${DST}' once the two files above exist"
+echo " * Verify each bumped file's version references look right (diff
against ${SRC})."
