> > The above is just for consideration if the robots.txt is ever > updated so the > > robots could be informed of this little detail. > > There was a push in '96 or '97 to update the robots.txt standard > and I > wrote a proposal back then > (http://www.conman.org/people/spc/robots2.html) > and while I still get the occasional email about it to my knowledge, > no > robot has implemented it (some portions perhaps, but not everything). > I > only mention this because it was attempted before.
Yes, that's the problem. Robot writers don't care for robots.txt improvements (it would just slow their robots down), and people in charge of maintaining robots.txt rarely/never hear about ideas discussed on this list. The push has to come from the latter group, I believe. I'm writing a simple little thing for indexing my Opera bookmarks which includes a crawler component, so if I have time I'll try implementing as much of the stuff mentioned in your proposal as I can, just for fun (some fun). Otis (where is my Turkey?) Gospodnetic __________________________________________________ Do You Yahoo!? Yahoo! GeoCities - quick and easy web site hosting, just $8.95/month. http://geocities.yahoo.com/ps/info1 -- This message was sent by the Internet robots and spiders discussion list ([EMAIL PROTECTED]). For list server commands, send "help" in the body of a message to "[EMAIL PROTECTED]".
