mån 2 nov. 2026 kl. 11:26 skrev Ivan Zhakov via dev < [email protected]>:
> On Mon, 12 Oct 2026 at 18:04, Nathan Hartman <[email protected]> > wrote: > >> On Fri, Oct 9, 2026 at 5:52 AM Evgeny Kotkov via dev >> <[email protected]> wrote: >> > >> > Hi all, >> > >> > I noticed that a Google search for "subversion 1.15" [1] currently >> shows the >> > staging copy of the release notes as the first result, at least for me: >> > >> > https://subversion-staging.apache.org/docs/release-notes/1.15.html >> > >> > It looks like our infra configuration for the staging subdomain does not >> > prevent search engines from indexing it. This is a problem for several >> > reasons: >> > >> > - Users may land on staging pages, which can be unfinished or out of >> > sync with the main site. >> > >> > - Search engines may incorrectly treat the pages on the main site as >> > duplicates of the staging ones. >> > >> > Could someone please ask infra to look into this? One option could be >> to add >> > a "X-Robots-Tag: noindex, nofollow" header for everything on the staging >> > subdomain. >> > >> > [1]: https://www.google.com/search?q=subversion+1.15 >> > >> > >> > Thanks, >> > Evgeny Kotkov >> >> >> Confirmed, and good catch! >> >> Perhaps this was caused by the release notes being in the staging site >> and not on the main site for an extended length of time. >> >> In any event, I agree that the staging site should not be indexed. >> >> I created Infra ticket 28512 for this [1]. >> >> [1] https://issues.apache.org/jira/browse/INFRA-28512 >> >> INFRA team suggested to use .htaccess. > > I have done the following: > 1. Added X-Robots-Tag=noindex, nofollow for sites where SERVER_NAME != > subversion.apache.org > 2. Added empty robots.txt for subversion.apache.org > 3. Added robots-noindex.txt and configured RewriteRule to use it as > robots.txt where SERVER_NAME != subversion.apache.org. > > So currently we have: > $ curl https://subversion.apache.org/robots.txt > [[[ > # Allow indexing by all User-Agent. > User-Agent: * > ]]] > > $ curl --head https://subversion.apache.org/ > [[[ > HTTP/1.1 200 OK > Connection: keep-alive > Content-Length: 14603 > Server: Apache > Access-Control-Allow-Origin: * > Content-Security-Policy: default-src 'self' data: blob: 'unsafe-inline' > 'unsafe-eval' https://www.apachecon.com/ > https://www.communityovercode.org/ https://*.apache.org/ > https://apache.org/ https://*.scarf.sh/ ; script-src 'self' data: blob: > 'unsafe-inline' 'unsafe-eval' https://www.apachecon.com/ > https://www.communityovercode.org/ https://*.apache.org/ > https://apache.org/ https://*.scarf.sh/ ; style-src 'self' data: blob: > 'unsafe-inline' 'unsafe-eval' https://www.apachecon.com/ > https://www.communityovercode.org/ https://*.apache.org/ > https://apache.org/ https://*.scarf.sh/ ; frame-ancestors 'self'; > frame-src 'self' data: blob: 'unsafe-inline' 'unsafe-eval' > https://www.apachecon.com/ https://www.communityovercode.org/ https://*. > apache.org/ https://apache.org/ https://*.scarf.sh/ ; worker-src 'self' > data: blob:; > Content-Type: text/html > Via: 1.1 varnish, 1.1 varnish > Accept-Ranges: bytes > Age: 0 > Date: Mon, 02 Nov 2026 10:10:22 GMT > X-Served-By: cache-hel1410027-HEL, cache-vie6362-VIE > X-Cache: HIT, MISS > X-Cache-Hits: 4, 0 > X-Timer: S1793614222.460933,VS0,VE37 > Vary: Accept-Encoding > Strict-Transport-Security: max-age=31536000; includeSubDomains; preload > ]]] > > $ curl https://subversion-staging.apache.org/robots.txt > [[[ > # Allow indexing by all User-Agent. > User-Agent: * > PS C:\Users\ivan> curl.exe > https://subversion-staging.apache.org/robots.txt > User-Agent: * > Disallow: / > > # Explicitly block some Google crawlers that ignore '*' rule [1] > # [1]: > https://developers.google.com/crawling/docs/crawlers-fetchers/google-special-case-crawlers > User-Agent: APIs-Google > Disallow: / > > User-Agent: AdsBot-Google-Mobile > Disallow: / > > User-Agent: AdsBot-Google > Disallow: / > > User-Agent: Mediapartners-Google > Disallow: / > ]]] > > $ curl --head https://subversion-staging.apache.org/ > [[[ > HTTP/1.1 200 OK > Connection: keep-alive > Content-Length: 14603 > Server: Apache > Access-Control-Allow-Origin: * > Content-Security-Policy: default-src 'self' data: blob: 'unsafe-inline' > 'unsafe-eval' https://www.apachecon.com/ > https://www.communityovercode.org/ https://*.apache.org/ > https://apache.org/ https://*.scarf.sh/ ; script-src 'self' data: blob: > 'unsafe-inline' 'unsafe-eval' https://www.apachecon.com/ > https://www.communityovercode.org/ https://*.apache.org/ > https://apache.org/ https://*.scarf.sh/ ; style-src 'self' data: blob: > 'unsafe-inline' 'unsafe-eval' https://www.apachecon.com/ > https://www.communityovercode.org/ https://*.apache.org/ > https://apache.org/ https://*.scarf.sh/ ; frame-ancestors 'self'; > frame-src 'self' data: blob: 'unsafe-inline' 'unsafe-eval' > https://www.apachecon.com/ https://www.communityovercode.org/ https://*. > apache.org/ https://apache.org/ https://*.scarf.sh/ ; worker-src 'self' > data: blob:; > X-Robots-Tag: noindex, nofollow > Content-Type: text/html > Via: 1.1 varnish, 1.1 varnish > Accept-Ranges: bytes > Age: 0 > Date: Mon, 02 Nov 2026 10:11:02 GMT > X-Served-By: cache-hel1410034-HEL, cache-vie6365-VIE > X-Cache: MISS, MISS > X-Cache-Hits: 0, 0 > X-Timer: S1793614262.087472,VS0,VE37 > Vary: Accept-Encoding > Strict-Transport-Security: max-age=31536000; includeSubDomains; preload > ]]] > > I think it's good enough. > > Thanks everyone for suggestions! > Thank you, LGTM! /Daniel >
