jiayuasu opened a new pull request, #3123: URL: https://github.com/apache/sedona/pull/3123
## Problem Old, versioned documentation is being indexed by search engines and ranking above the current docs. For example, searching *"sedona apache discord"* returns the outdated `1.4.1` Discord invite page as the top result. Root cause, verified against the live site: | Check | Result | |---|---| | `https://sedona.apache.org/1.4.1/community/discord-invite-form.html` | **HTTP 200** — live & crawlable | | Its `rel=canonical` → `https://sedona.apache.org/latest/community/discord-invite-form/` | **HTTP 404** — page no longer exists in current docs | | `robots` meta tag on the page | none | The `mike` plugin already sets `canonical_version: latest`, so every archived page carries a canonical to its `/latest/` equivalent. But when that equivalent has been removed (as with the Discord invite page), the canonical points at a 404 — search engines then ignore it and index the stale archived page. Archived numbered versions are frozen once released and never rebuilt, so a build-time `noindex` in the theme template cannot reach the pages that are already deployed and indexed. The only control that covers all previously-published versions at once is a site-root `robots.txt`. ## Change Add a root-level `robots.txt`, served at `https://sedona.apache.org/robots.txt`: - Keeps only the current stable docs (`/latest/`) crawlable. - Blocks the development snapshot (`/latest-snapshot/`) and every numbered archived version. - Numbered versions all live in a top-level directory beginning with a digit, so blocking the digit prefixes (`/0`–`/9`) covers all past **and future** releases with no per-release maintenance. This also blocks the numbered duplicate of the current release (e.g. `/1.9.0/`) while keeping `/latest/` indexed — clean deduplication. - Leaves the separate `/sedonadb/` and `/spatialbench/` sub-sites on the same domain untouched. - Advertises the current sitemap (`/latest/sitemap.xml`). This file sits alongside the other hand-maintained root files on the `website` branch (`.asf.yaml`, `.nojekyll`); `mike` preserves root-level files it does not manage when deploying new versions. ## Validation Checked the rules against a robots parser for the URL classes that matter: - Allowed: `/latest/…`, `/latest/sitemap.xml`, `/sedonadb/…`, `/spatialbench/…`, `/`, `/versions.json` - Blocked: `/latest-snapshot/…`, `/1.4.1/…`, `/1.9.0/…`, `/1.0.0-1.2.0-incubating/…` ## Note `robots.txt` stops crawling and drops stale pages from results over time; it does not force instant removal of already-indexed URLs. Removal can be accelerated via the Search Console removal tool if desired. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
