On 17 Jan 2001, at 15:23, Mark Roberts wrote:
> Most legitimate search engine spiders (and I'll count Lycos among these,
> regardless of what I think about their harveting my images and keeping local
> copies) will follow the robots.txt file in your root directory. I don't know
> about meta tags, though. Non-legitimate harvesters (spambot address harvesters
> and the like) do whatever they damn well feel like.
Lycos claim that they adhere to the meta exclusion standards however it
doesn't appear that they utilise the NOIMAGEINDEX tag, very convenient.
This tag allows all text to be indexed but disallows images.
Unfortunately utilising the robot.txt file in the root of the server is quite
limited, you can disallow harvesting in a particular directory however the way
the that the PUG site is assembled the image files are held in the same
directories as the html pages so limited information at the site would be
indexed. Unless of course the search bot totally ignores the request, as you
pointed out :-(
Cheers,
Rob Studdert
HURSTVILLE AUSTRALIA
Tel +61-2-9554-4110
Fax +61-2-9554-9259
UTC(GMT) +10 Hours
[EMAIL PROTECTED]
http://www.ozemail.com.au/~distudio/publications.html
-
This message is from the Pentax-Discuss Mail List. To unsubscribe,
go to http://www.pdml.net and follow the directions. Don't forget to
visit the Pentax Users' Gallery at http://pug.komkon.org.
