Caffeinate The World wrote:
>
> if i insert a bunch of URLs into the db via a text file.. and those
> urls have no corresponding server command, by default, it should have a
> Follow of path and NOT site.
>
> here is one example. i pull URLs from a sub section of dmoz using the
> Href directive. Some of those are site URLs and some are a path of a
> site. now, when i pull all the urls from that sub section of dmoz, the
> current setting will index thru all the "whole" sites/servers instead
> of just the paths.
>
> if default is to follow path, then the whole site would still get
> indexed. i hope that made sense. i brought this topic up about a month
> ago and i still see this behavior in -current.
When "DeleteNoServer no" is specified, indexer follows whole site
if there is no "Server" command for some URL.
___________________________________________
If you want to unsubscribe send "unsubscribe general"
to [EMAIL PROTECTED]
