When do you use a robots.txt file?
Ken: I've personally been evangelizing the use of robots.txt for controlling
access to RSS feeds. Some publishers want to offer full-text feeds to
individual users, but don't want that full-text harvested by robots and
republished... robots.txt is the best
If that data is never served directly and only programmatically, then it
should probably not even be under the web root.
--- Ben
-Original Message-
From: Ken Ketsdever [mailto:[EMAIL PROTECTED]
Sent: Monday, January 09, 2006 4:32 PM
To: CF-Talk
Subject: Robots.txt - - best practices
, 2006 1:37 PM
To: CF-Talk
Subject: RE: Robots.txt - - best practices
If that data is never served directly and only programmatically, then it
should probably not even be under the web root.
--- Ben
-Original Message-
From: Ken Ketsdever [mailto:[EMAIL PROTECTED]
Sent: Monday, January
robots.txt is intended as a mechanism that directs the good guys on how
to index your site (the search engines that want to follow the rules).
Personally, I wouldn't use robots.txt as a way to hide a directory,
although you still might want to tell the SEs to ignore it.
-Original
http://www.freefind.com/library/howto/robots/
The robots.txt is really to control how your page is indexed by search
engines. This file should not be used for access control nor security,
and you can't even guarantee that search engines obey the file. Most
respectable spider's follow that
Would that same rule of thumb apply for logic code? Say if I
separate out all of my queries, cfc's etc... from the
presentation layer should I keep those logic based templates
out of the root as well?
Yes, to the extent that this is practical. For example, I typically place
CFCs, custom
that trouble why not just stick the CFC's in a web
accessible place to begin with?
~Brad
-Original Message-
From: Dave Watts [mailto:[EMAIL PROTECTED]
Sent: Monday, January 09, 2006 3:58 PM
To: CF-Talk
Subject: RE: Robots.txt - - best practices
Would that same rule of thumb apply for logic
(that includes
images, JavaScript files, CSS files, etc.) needs to be under the web root.
Not much else does, and therefore should not be there.
--- Ben
-Original Message-
From: Dave Watts [mailto:[EMAIL PROTECTED]
Sent: Monday, January 09, 2006 4:58 PM
To: CF-Talk
Subject: RE: Robots.txt
The problem I am having now is that the CFC's I use with
flash remoting have to be web accessible, which screws up the
standard I use for keeping my CFC's outside the WWW root.
Anybody have any ides. I thought about a wrapper CFC that
just called the real CFC, but if I am going to go
to go through that trouble why
not just stick the CFC's in a web accessible place to begin with?
~Brad
-Original Message-
From: Dave Watts [mailto:[EMAIL PROTECTED]
Sent: Monday, January 09, 2006 3:58 PM
To: CF-Talk
Subject: RE: Robots.txt - - best practices
Would that same rule of thumb
Mapping to a directory outside of the webroot is the same as having the
files in the webroot. If you can access the file from a URL it's public
domain.
One of the most dangerous file in any application is going to be
application.cfm we all know were that's at...
Casey
From: Dave
Yes, to
Mapping to a directory outside of the webroot is the same as having the files
in the webroot. If you can access the file from a URL it's public domain.
IIS mapping, yes this is true. ColdFusion mapping, no people can not.
--
Ian Skinner
Web Programmer
BloodSource
Message-
From: Matthew Walker [mailto:[EMAIL PROTECTED]
Sent: Monday, January 09, 2006 4:20 PM
To: CF-Talk
Subject: RE: Robots.txt - - best practices
Yeah that's what we do -- create a façade CFC just for Flash that calls the
regular CFC. CFCs aren't really about saving effort
So, somebody send me an example of what their CFC façade
looks like. Do you have to code a new method into the façade
every time you add a cfc, or is it completely dynamic where
you pass in the name of the cfc and the method you want etc
and your façade never has to change?
In the few
Mapping to a directory outside of the webroot is the same as
having the files in the webroot. If you can access the file
from a URL it's public domain.
One of the most dangerous file in any application is going to
be application.cfm we all know were that's at...
This only applies to web
Source code viewing exploits do not allow you to view anything that isn't
accessible through the web server.
I see. I assumed a code viewing exploit would be where someone was able to
execute a cffile or cfdirectory. On a lot of servers you can see anything on
the C: drive with the
Thanks Dave, I though that was what you were refering to. For a second there
I though you were refering to setting up Mappings in CF Administrator. Yeah
I love almost blank Application.cfm files. Of course coldfusion is looking
for it so might as well use it for something include
What about invoking the CFC? If someone could invoke a
web-accessible CFC, couldn't they invoke it all the same
through the façade?
Sure. However, you could write additional code within the façade to ensure
that it's only being invoked in specific ways, such as a web service call,
using the
When do you use a robots.txt file?
What directories do you disallow?
In my case mainly two things:
1. normally, ie. to tell robots pages not worth indexing, mainly because
the require some sort of login
that robots won't be able to pass anyway;
2. I've declared a trap template in robots.txt
19 matches
Mail list logo
