anatoly techtonik writes:
> cc: tracker-discuss for comments, please reply to infra
> *Hi,I'd like to patch http://bugs.python.org/robots.txt with:
>
> Disallow: /msg*
>
> This should prevent indexing of individual messages, which are
> included on bug pages anyway.
It might work that way, but it shouldn't. From the Robot Exclusion
Standard:
Disallow
The value of this field specifies a partial URL that is not to be
visited. This can be a full path, or a partial path; any URL that
starts with this value will not be retrieved. For example, Disallow:
/help disallows both /help.html and /help/index.html[.]
So just
Disallow: /msg
is presumably what you want.
_______________________________________________
Tracker-discuss mailing list
[email protected]
https://mail.python.org/mailman/listinfo/tracker-discuss
