Author: ivan
Date: Fri Oct 30 16:08:09 2026
New Revision: 1939750

Log:
In site/staging: 

Explicitly block some Google crawlers that ignore '*' rule [1]

[1]: 
https://developers.google.com/crawling/docs/crawlers-fetchers/google-special-case-crawlers

* robots-noindex.txt: Add explicit User-Agent sections.

Modified:
   subversion/site/staging/robots-noindex.txt

Modified: subversion/site/staging/robots-noindex.txt
==============================================================================
--- subversion/site/staging/robots-noindex.txt  Fri Oct 30 16:04:11 2026        
(r1939749)
+++ subversion/site/staging/robots-noindex.txt  Fri Oct 30 16:08:09 2026        
(r1939750)
@@ -1,2 +1,25 @@
 User-Agent: *
 Disallow: /
+
+# Explicitly block some Google crawlers that ignore '*' rule [1]
+# [1]: 
https://developers.google.com/crawling/docs/crawlers-fetchers/google-special-case-crawlers
+User-Agent: APIs-Google
+Disallow: /
+
+User-Agent: AdsBot-Google-Mobile
+Disallow: /
+
+User-Agent: AdsBot-Google-Mobile
+Disallow: /
+
+User-Agent: AdsBot-Google-Mobile
+Disallow: /
+
+User-Agent: AdsBot-Google-Mobile
+Disallow: /
+
+User-Agent: AdsBot-Google-Mobile
+Disallow: /
+
+User-Agent: AdsBot-Google-Mobile
+Disallow: /

Reply via email to