I'm inclined to agree that a second file would probably get overlooked
by bots. I would imagine it was difficult trying to get those who run them
to respect the first one.
I was unaware of the 'Allow' command. Is there a URL that documents it?
Also, the use of wildcards when giving paths should be incorporated.
That would greatly reduce the number of path lines that you'd have to type
into and a robot would have to read out of the robots.txt file. And,
wildcards shouldn't be limited to just the end of the path. You should be
able to use them in the middle as well. Perhaps the UNIX brackets could be
incorporated ( [0-9]. [a,e.i.o.u]) as matching characters.
As far as the number of sites that actually use robots.txt, that would
grow as the strength of the robots.txt coding improved.
Fred
----- Original Message -----
From: "Paul Trippett" <[EMAIL PROTECTED]>
To: "'Internet robots, spiders, web-walkers, etc.'" <[EMAIL PROTECTED]>
Sent: Sunday, January 11, 2004 8:57 AM
Subject: RE: [Robots] Robots.txt Evolution?
>
> > This would need to be a separate file, probably "robots2.txt".
>
> People who are oblivious to the robots.txt standard already complain
> about grabbing robots.txt and would thought that getting another file
> would make them complain a little more. Besides who wants to maintain
> two files that do the same job. In my view, if it was in another file an
> extended standard would never be used and then yes, there would be no
> point.
>
> /pt
>
>
> _______________________________________________
> Robots mailing list
> [EMAIL PROTECTED]
> http://www.mccmedia.com/mailman/listinfo/robots
_______________________________________________
Robots mailing list
[EMAIL PROTECTED]
http://www.mccmedia.com/mailman/listinfo/robots
