Author: ivan
Date: Mon Nov  2 10:05:07 2026
New Revision: 1939942

Log:
In site/publish:

Merge the outstanding changes from staging (Add robots-noindex.txt for staging
branches).

Added:
   subversion/site/publish/robots-noindex.txt
      - copied unchanged from r1939941, 
subversion/site/staging/robots-noindex.txt
Modified:
   subversion/site/publish/   (props changed)
   subversion/site/publish/.htaccess

Modified: subversion/site/publish/.htaccess
==============================================================================
--- subversion/site/publish/.htaccess   Mon Nov  2 09:40:12 2026        
(r1939941)
+++ subversion/site/publish/.htaccess   Mon Nov  2 10:05:07 2026        
(r1939942)
@@ -19,3 +19,8 @@ RewriteRule ^/?download(\.html)?/?$ /dow
 <If "%{SERVER_NAME} != 'subversion.apache.org'">
     Header set X-Robots-Tag "noindex, nofollow"
 </If>
+
+# Use robots-noindex.txt for /robots.txt for all VirtualHost except
+# the primary one to exclude them from indexing.
+RewriteCond expr "%{SERVER_NAME} != 'subversion.apache.org'"
+RewriteRule ^robots.txt(\?.*)?$ robots-noindex.txt$1 [PT]

Copied: subversion/site/publish/robots-noindex.txt (from r1939941, 
subversion/site/staging/robots-noindex.txt)
==============================================================================
--- /dev/null   00:00:00 1970   (empty, because file is newly added)
+++ subversion/site/publish/robots-noindex.txt  Mon Nov  2 10:05:07 2026        
(r1939942, copy of r1939941, subversion/site/staging/robots-noindex.txt)
@@ -0,0 +1,16 @@
+User-Agent: *
+Disallow: /
+
+# Explicitly block some Google crawlers that ignore '*' rule [1]
+# [1]: 
https://developers.google.com/crawling/docs/crawlers-fetchers/google-special-case-crawlers
+User-Agent: APIs-Google
+Disallow: /
+
+User-Agent: AdsBot-Google-Mobile
+Disallow: /
+
+User-Agent: AdsBot-Google
+Disallow: /
+
+User-Agent: Mediapartners-Google
+Disallow: /

Reply via email to