I'm forever creating "test" web pages that aren't linked to, so that our
Intranet spider doesn't find them and index them. (Security through
obscurity). But recently I found that some of them were in our search
engine, and it turned out that an Analog report had been posted and
indexed, and the spider followed the links in the request report to find
my "test" pages.
While I immediately added the directory that the Analog report was in to
the servers ROBOTS.TXT file, it would be handy if Analog could
automatically add the Robot Exclusion meta tag to its output. (Now that
I've made anlgform.exe available on the server, I can't always keep
track where an Analog report will be posted. (And I don't have the tools
to modify and recompile the code with the addition).
The Robots exclusion tag looks like this:
<META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW">
It's explained in greater detail at:
http://info.webcrawler.com/mak/projects/robots/exclusion.html#meta
Aengus
--------------------------------------------------------------------
This is the analog-help mailing list. To unsubscribe from this
mailing list, send mail to [EMAIL PROTECTED]
with "unsubscribe analog-help" in the main BODY OF THE MESSAGE.
--------------------------------------------------------------------
