Author: Tim Hewitt
Email: [EMAIL PROTECTED]
Message:
Oops. I found another hole it appears in the robots.txt adherance.
According to the spec I should be able to do something like this:
User-agent: *
Allow: /forumdisplay.php
Allow: /announcement.php
Allow: /showthread.php
Disallow: /
and as a result, only the top three files would be indexed - everything else in the
site would be excluded.
This does not work. The Allow directive is apparently being ignored - or simply not
treated as a prefix like the previous problem? The actual URLs are again of the format
forumdisplay.php?forumid=9, etc.
Thanks for looking at this.
-Tim
Reply: <http://search.mnogo.ru/board/message.php?id=2073>
___________________________________________
If you want to unsubscribe send "unsubscribe general"
to [EMAIL PROTECTED]
