jiayuasu opened a new pull request, #3125: URL: https://github.com/apache/sedona/pull/3125
## Did you read the Contributor Guide? - Yes, I have read the [Contributor Rules](https://sedona.apache.org/latest/community/rule/) and [Contributor Development Guide](https://sedona.apache.org/latest/community/develop/) ## Is this PR related to a ticket? - No: - this is a documentation update. The PR name follows the format `[DOCS] my subject` ## What changes were proposed in this PR? Old, versioned documentation is being indexed by search engines and ranking above the current docs. For example, searching *"sedona apache discord"* returns the outdated `1.4.1` Discord invite page as the top result. Root cause, verified on the live site: | Check | Result | |---|---| | `https://sedona.apache.org/1.4.1/community/discord-invite-form.html` | **HTTP 200** — live & crawlable | | Its `rel=canonical` → `https://sedona.apache.org/latest/community/discord-invite-form/` | **HTTP 404** — removed from current docs | | `robots` meta tag on the page | none | `mike` already sets `canonical_version: latest`, so archived pages carry a canonical to their `/latest/` equivalent — but when that equivalent has been removed, the canonical points at a 404 and search engines ignore it and index the stale archived page. The intuitive fix — a `robots.txt` `Disallow` for the archived paths — does **not** work here: a disallowed page can no longer be crawled, so search engines never observe a removal signal and may keep the stale URL in the index (Google explicitly advises against robots.txt for removal, and Bing similarly needs the page crawlable to see a `noindex`). The reliable signal on static hosting (no control over response headers / redirects / 410s) is a **crawlable** `<meta name="robots" content="noindex">`. This PR adds `tools/noindex_archived_docs.py` and a post-deploy step in `docs.yml` that: - Adds `<meta name="robots" content="noindex, follow">` to every page of the archived versions and the development snapshot on the published `website` branch, leaving them crawlable so search engines can drop them. - **Skips the current stable version.** `latest` is a symlink to that version's directory, so tagging it would de-index the live docs; its duplicate versioned URL (e.g. `/1.9.0/`) is already consolidated onto `/latest/` by the existing canonical tag. - Handles raw HTML fragments with no `<head>` (the Discord invite pages in `1.4.1`/`1.5.0` — exactly the reported case) by prepending the tag, which the HTML parser hoists into the head. - Writes a sitemap-only `robots.txt` (advertises `/latest/sitemap.xml`, blocks nothing) so the `noindex` stays crawlable. - Is idempotent: pages that already carry a robots meta tag are untouched. It **backfills** existing archived versions on the next deploy and tags each version automatically as a new release supersedes it — no per-release maintenance. Supersedes #3123 (the earlier robots.txt-only approach, which had this exact removal limitation). ## How was this patch tested? - Ran the script against a current snapshot of the `website` branch: **19,671** archived pages tagged across all numbered versions + `latest-snapshot`; the current stable `1.9.0` (and the `latest` symlink) left untouched; **0** archived HTML pages left without a robots meta, including both headless Discord fragment pages. - Verified idempotency (a second run tags 0 pages) and unit-level behavior in sandboxes (current skipped, archived + headless tagged, non-HTML ignored). - `tools/noindex_archived_docs.py` passes `black`, `pyupgrade`, and the license-header hook; `docs.yml` passes `actionlint` and `yamllint` (pre-commit). ## Did this PR include necessary documentation updates? - No, this PR does not affect any public API so no need to change the documentation. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
