Hi,
I've just subscribed to the robots.txt list. I read the messages posted
through December.
My question is if there is going to be any evolution to the robots.txt
coding? It is very limited at present. I can think of a few things they
could incorporate that would make it bettter and I'm sure the rest of you
could, too.
I've had robots.txt files on my sites for years. When I recently
researched to see what had changed, it doesn't appear that there is anything
new on the horizon.
I've got two robots completely blocked out of my system. One is
Scooter, which is Alta Vista's robot. When I initially put it in as
disallowed, it was because Scooter was hitting my site several times a day.
I don't mind them scanning me once in a while to get listings for search
engines, but I do object to them hammering my site that frequently.
Should not there be coding to tell either a particular robot or group of
robots how often they are allowed to scan my site? Maybe a line like (and
this is arbitrary):
User-agent: Scooter
Interval: 30d
Disallow: /whatever your want them not to scan when they do come in.
This would tell Scooter that he is not to scan again until thirty days
after the last scan. There could be codes like h[ours], d[ays], w[eeks],
m[onths], and y[ears].
Just an idea I had.
Feedback?
Fred
_______________________________________________
Robots mailing list
[EMAIL PROTECTED]
http://www.mccmedia.com/mailman/listinfo/robots
