On Mon, 12 Oct 2026 at 18:04, Nathan Hartman <[email protected]> wrote:
> On Fri, Oct 9, 2026 at 5:52 AM Evgeny Kotkov via dev > <[email protected]> wrote: > > > > Hi all, > > > > I noticed that a Google search for "subversion 1.15" [1] currently shows > the > > staging copy of the release notes as the first result, at least for me: > > > > https://subversion-staging.apache.org/docs/release-notes/1.15.html > > > > It looks like our infra configuration for the staging subdomain does not > > prevent search engines from indexing it. This is a problem for several > > reasons: > > > > - Users may land on staging pages, which can be unfinished or out of > > sync with the main site. > > > > - Search engines may incorrectly treat the pages on the main site as > > duplicates of the staging ones. > > > > Could someone please ask infra to look into this? One option could be > to add > > a "X-Robots-Tag: noindex, nofollow" header for everything on the staging > > subdomain. > > > > [1]: https://www.google.com/search?q=subversion+1.15 > > > > > > Thanks, > > Evgeny Kotkov > > > Confirmed, and good catch! > > Perhaps this was caused by the release notes being in the staging site > and not on the main site for an extended length of time. > > In any event, I agree that the staging site should not be indexed. > > I created Infra ticket 28512 for this [1]. > > [1] https://issues.apache.org/jira/browse/INFRA-28512 > > INFRA team suggested to use .htaccess. I have done the following: 1. Added X-Robots-Tag=noindex, nofollow for sites where SERVER_NAME != subversion.apache.org 2. Added empty robots.txt for subversion.apache.org 3. Added robots-noindex.txt and configured RewriteRule to use it as robots.txt where SERVER_NAME != subversion.apache.org. So currently we have: $ curl https://subversion.apache.org/robots.txt [[[ # Allow indexing by all User-Agent. User-Agent: * ]]] $ curl --head https://subversion.apache.org/ [[[ HTTP/1.1 200 OK Connection: keep-alive Content-Length: 14603 Server: Apache Access-Control-Allow-Origin: * Content-Security-Policy: default-src 'self' data: blob: 'unsafe-inline' 'unsafe-eval' https://www.apachecon.com/ https://www.communityovercode.org/ https://*.apache.org/ https://apache.org/ https://*.scarf.sh/ ; script-src 'self' data: blob: 'unsafe-inline' 'unsafe-eval' https://www.apachecon.com/ https://www.communityovercode.org/ https://*.apache.org/ https://apache.org/ https://*.scarf.sh/ ; style-src 'self' data: blob: 'unsafe-inline' 'unsafe-eval' https://www.apachecon.com/ https://www.communityovercode.org/ https://*.apache.org/ https://apache.org/ https://*.scarf.sh/ ; frame-ancestors 'self'; frame-src 'self' data: blob: 'unsafe-inline' 'unsafe-eval' https://www.apachecon.com/ https://www.communityovercode.org/ https://*.apache.org/ https://apache.org/ https://*.scarf.sh/ ; worker-src 'self' data: blob:; Content-Type: text/html Via: 1.1 varnish, 1.1 varnish Accept-Ranges: bytes Age: 0 Date: Mon, 02 Nov 2026 10:10:22 GMT X-Served-By: cache-hel1410027-HEL, cache-vie6362-VIE X-Cache: HIT, MISS X-Cache-Hits: 4, 0 X-Timer: S1793614222.460933,VS0,VE37 Vary: Accept-Encoding Strict-Transport-Security: max-age=31536000; includeSubDomains; preload ]]] $ curl https://subversion-staging.apache.org/robots.txt [[[ # Allow indexing by all User-Agent. User-Agent: * PS C:\Users\ivan> curl.exe https://subversion-staging.apache.org/robots.txt User-Agent: * Disallow: / # Explicitly block some Google crawlers that ignore '*' rule [1] # [1]: https://developers.google.com/crawling/docs/crawlers-fetchers/google-special-case-crawlers User-Agent: APIs-Google Disallow: / User-Agent: AdsBot-Google-Mobile Disallow: / User-Agent: AdsBot-Google Disallow: / User-Agent: Mediapartners-Google Disallow: / ]]] $ curl --head https://subversion-staging.apache.org/ [[[ HTTP/1.1 200 OK Connection: keep-alive Content-Length: 14603 Server: Apache Access-Control-Allow-Origin: * Content-Security-Policy: default-src 'self' data: blob: 'unsafe-inline' 'unsafe-eval' https://www.apachecon.com/ https://www.communityovercode.org/ https://*.apache.org/ https://apache.org/ https://*.scarf.sh/ ; script-src 'self' data: blob: 'unsafe-inline' 'unsafe-eval' https://www.apachecon.com/ https://www.communityovercode.org/ https://*.apache.org/ https://apache.org/ https://*.scarf.sh/ ; style-src 'self' data: blob: 'unsafe-inline' 'unsafe-eval' https://www.apachecon.com/ https://www.communityovercode.org/ https://*.apache.org/ https://apache.org/ https://*.scarf.sh/ ; frame-ancestors 'self'; frame-src 'self' data: blob: 'unsafe-inline' 'unsafe-eval' https://www.apachecon.com/ https://www.communityovercode.org/ https://*.apache.org/ https://apache.org/ https://*.scarf.sh/ ; worker-src 'self' data: blob:; X-Robots-Tag: noindex, nofollow Content-Type: text/html Via: 1.1 varnish, 1.1 varnish Accept-Ranges: bytes Age: 0 Date: Mon, 02 Nov 2026 10:11:02 GMT X-Served-By: cache-hel1410034-HEL, cache-vie6365-VIE X-Cache: MISS, MISS X-Cache-Hits: 0, 0 X-Timer: S1793614262.087472,VS0,VE37 Vary: Accept-Encoding Strict-Transport-Security: max-age=31536000; includeSubDomains; preload ]]] I think it's good enough. Thanks everyone for suggestions! -- Ivan Zhakov
