I thought about using the meta tag for
robots not to follow or index the site, but
realized I'd have to take that out of the
file every time I uploaded the site to my
client's domain.
Use robots.txt:
http://www.robotstxt.org/
Dave Watts, CTO, Fig Leaf Software
Put Apache/IIS authentication on your dev sites. (Of course much easier in
apache)
Russ
-Original Message-
From: Claude Schneegans [mailto:[EMAIL PROTECTED]
Sent: Wednesday, February 13, 2008 12:09 PM
To: CF-Talk
Subject: Re: OT: How to stop bots indexing my dev sites
Use
: How to stop bots indexing my dev sites
Use robots.txt:
This is correct to control good bots, but bad bots don't read it.
Some (very) bad bots even exploit what is in the file to read what's
forbidden.
~|
Adobe® ColdFusion® 8
Use robots.txt:
This is correct to control good bots, but bad bots don't read it.
Some (very) bad bots even exploit what is in the file to read what's
forbidden.
~|
Adobe® ColdFusion® 8 software 8 is the most important and
Once I started reviewing rankings, I found that my development
site was ranking higher for the content than my client's main site!
Not good!
Exact.
But robot control is not something trivial.
Firstly, there are true and friendly robots, like Google, secondly,
there are bad bots, looking for
If he is simply
trying to prevent search engines from indexing certain folders, he should be
fine?
Let's say 50% fine, since bad bots (may be 50% of all robot traffic) do
not even care reading robots.txt
~|
Adobe® ColdFusion®
Use robots.txt:
This is correct to control good bots, but
bad bots don't read it.
This isn't relevant, since the crawlers used by Google and Yahoo do obey
robots.txt.
Some (very) bad bots even exploit what
is in the file to read what's forbidden.
Robots.txt is not intended to be a
We restrict the dev site/server by IP address, so only local
connections are permitted by the webserver. VPN access is used to
access a dev server off-site.
On Feb 14, 2008 12:56 AM, Rick Faircloth [EMAIL PROTECTED] wrote:
Hi, all.
I usually put up a client's site while it is development
so
-Talk
Subject: Re: OT: How to stop bots indexing my dev sites
We restrict the dev site/server by IP address, so only local
connections are permitted by the webserver. VPN access is used to
access a dev server off-site.
On Feb 14, 2008 12:56 AM, Rick Faircloth [EMAIL PROTECTED] wrote:
Hi
to stop bots indexing my dev sites
We restrict the dev site/server by IP address, so only local
connections are permitted by the webserver. VPN access is used to
access a dev server off-site.
On Feb 14, 2008 12:56 AM, Rick Faircloth [EMAIL PROTECTED] wrote:
Hi, all.
I usually put up
10 matches
Mail list logo
