At 5:31 PM -0400 9/14/98, Michael Graff wrote:
>There is a way to limit what pages are indexed in htdig, using meta
>tags. I propose an additional flag that says "collect URLs from this
>page, but do NOT index this page."
Sorry, the robots exlusion standard people (and myself) are ahead of you.
In version 3.1.0b1, htdig accepts the following tag (check
http://www.htdig.org/meta.html for details)
<meta name="robots" content="noindex">
This means exactly as you say. Follow links from the page but don't index
them. There's also <meta name="robots" content="nofollow"> for those pages
that you don't want the URLs followed. :-)
-Geoff Hutchison
Williams Students Online
http://wso.williams.edu/
----------------------------------------------------------------------
To unsubscribe from the htdig mailing list, send a message to
[EMAIL PROTECTED] containing the single word "unsubscribe" in
the body of the message.
