Author: ivan
Date: Sat Oct 31 15:08:21 2026
New Revision: 1939806

Log:
In site/staging: Follow-up to r1939750: Fix list of Google crawlers that
ignore '*' rule.

Found by: dsahlberg

* robots-noindex.txt: Add AdsBot-Google and Mediapartners-Google User-Agent
  rules. Remove duplicate address for AdsBot-Google-Mobile.

Modified:
   subversion/site/staging/robots-noindex.txt

Modified: subversion/site/staging/robots-noindex.txt
==============================================================================
--- subversion/site/staging/robots-noindex.txt  Sat Oct 31 14:43:30 2026        
(r1939805)
+++ subversion/site/staging/robots-noindex.txt  Sat Oct 31 15:08:21 2026        
(r1939806)
@@ -9,17 +9,8 @@ Disallow: /
 User-Agent: AdsBot-Google-Mobile
 Disallow: /
 
-User-Agent: AdsBot-Google-Mobile
-Disallow: /
-
-User-Agent: AdsBot-Google-Mobile
+User-Agent: AdsBot-Google
 Disallow: /
 
-User-Agent: AdsBot-Google-Mobile
-Disallow: /
-
-User-Agent: AdsBot-Google-Mobile
-Disallow: /
-
-User-Agent: AdsBot-Google-Mobile
+User-Agent: Mediapartners-Google
 Disallow: /

Reply via email to