Author: Alexander Barkov
Email: [EMAIL PROTECTED]
Message:
Hi!
Robots.txt support was implemented according to this document
http://www.robotstxt.org/wc/norobots.html
We'll implement Allow commands in 3.2.branch. Thanks for
suggestion.
> Oops. I found another hole it appears in the robots.txt adherance.
>
> According to the spec I should be able to do something like this:
>
> User-agent: *
> Allow: /forumdisplay.php
> Allow: /announcement.php
> Allow: /showthread.php
> Disallow: /
>
> and as a result, only the top three files would be indexed - everything else in the
>site would be excluded.
>
> This does not work. The Allow directive is apparently being ignored - or simply not
>treated as a prefix like the previous problem? The actual URLs are again of the
>format forumdisplay.php?forumid=9, etc.
>
> Thanks for looking at this.
>
> -Tim
>
>
Reply: <http://search.mnogo.ru/board/message.php?id=2090>
___________________________________________
If you want to unsubscribe send "unsubscribe general"
to [EMAIL PROTECTED]
