Thanks for the URL. We do have a robots file in the root of each site. Perhaps the meta tags will help. I'll check it
out.
-Mark
-----Original Message-----
From: Stephen Moretti [mailto:[EMAIL PROTECTED]
Sent: Monday, April 05, 2004 8:32 AM
To: CF-Talk
Subject: Re: user agent checking and spidering...
Mark A. Kruger - CFG wrote:
> Dave,
>
> That's not what I'm finding. If you have a robots.txt file that says:
>
> disallow /search.cfm
>
> It will not index the search.cfm file from the root of the server. But
> I cannot find anywhere where you can put in
> something like this:
>
> disallow http://www.someothersite.com
>
> You see what I mean? The robot.txt file allows you to exclude pages on
> THIS site that you don't want indexed.
>
You'll need to do one robots.txt file in the root of each site you want
to control robot access to. You cannot prevent a robot from spidering
another site, but generally the robot will only be working on one URI at
a time otherwise it would never complete a task.
Check out http://www.robotstxt.org/wc/robots.html for details on the
robots exclusion standard. There is also some stuff in there regarding
meta tags for robot exclusion.
Hope this helps
Regards
Stephen
[Todays Threads] [This Message] [Subscription] [Fast Unsubscribe] [User Settings]
