Search Engines and Files
I keep a close eye on my web logs and saw that my CV was very popular, to my surprise! I was thinking “why in gods name do they want to download my CV that many times?”, I would have thought nothing of it was only like once a week or so, but more than 3 or 4 times a day was a bit strange in my books.
So I started my investigation, and followed the referrers, one of them was
You can’t really blame anyone for clicking a link like
COLDFUSION / SQL
/ (MICROSOFT WORD)
... Author: Ben
Forta (QUE); ColdFusion 4.0 Author: Ben ... Mastering
ColdFusion 4.5 Author:
Arman Danesh & ... Mastering
Regular Expressions ** Author: Jeffrey EF Freidl (O
...
www.tacofleur.com/file/public/TacoFleurCV.doc - View as
html
So the file gateway I created was not sufficient,
with the file gateway I was trying to guide people before they download a file
from my website, i.e. given them a bit more information about the file and
making sure they understand they are going to download a file of X bytes.
However the search engine spiders picked up my file directly, and with entries
like the above in the search engine I suspect a lot of wasted bandwidth will
have to be dealt with.
The next best thing I could come up with was to
create 3 sub directories under the “File” directory, respectively
being:
file/public/
file/private/
file/list/
and create a robots.txt file with
the following entries
----------------------
User-agent: *
Disallow: /file/private/ # Files should not be indexed in the search engine
Disallow: /file/public/ # Files should not be indexed in the search engine
----------------------
The only downfall is that I now have to duplicate the files I want indexed, the only other alternative is to create separate entries for each file I do not want indexed.
If anyone has a better idea I’d love to hear about it..
Tell me and I will forget
Show me and I will remember
Teach me and I will learn
<<clip_image001.gif>>
--- You are currently subscribed to cfaussie as: [EMAIL PROTECTED] To unsubscribe send a blank email to [EMAIL PROTECTED]
MXDU2004 + Macromedia DevCon AsiaPac + Sydney, Australia http://www.mxdu.com/ + 24-25 February, 2004
