Author: ivan Date: Fri Oct 30 16:08:09 2026 New Revision: 1939750 Log: In site/staging:
Explicitly block some Google crawlers that ignore '*' rule [1] [1]: https://developers.google.com/crawling/docs/crawlers-fetchers/google-special-case-crawlers * robots-noindex.txt: Add explicit User-Agent sections. Modified: subversion/site/staging/robots-noindex.txt Modified: subversion/site/staging/robots-noindex.txt ============================================================================== --- subversion/site/staging/robots-noindex.txt Fri Oct 30 16:04:11 2026 (r1939749) +++ subversion/site/staging/robots-noindex.txt Fri Oct 30 16:08:09 2026 (r1939750) @@ -1,2 +1,25 @@ User-Agent: * Disallow: / + +# Explicitly block some Google crawlers that ignore '*' rule [1] +# [1]: https://developers.google.com/crawling/docs/crawlers-fetchers/google-special-case-crawlers +User-Agent: APIs-Google +Disallow: / + +User-Agent: AdsBot-Google-Mobile +Disallow: / + +User-Agent: AdsBot-Google-Mobile +Disallow: / + +User-Agent: AdsBot-Google-Mobile +Disallow: / + +User-Agent: AdsBot-Google-Mobile +Disallow: / + +User-Agent: AdsBot-Google-Mobile +Disallow: / + +User-Agent: AdsBot-Google-Mobile +Disallow: /
