It was thus said that the Great Paul Trippett once stated:
>
> As the Original Robots.txt standard is 7 years old I agree it is time
> for a review and maybe extending the functionality of the robots.txt
> file.
I think it's a bit older than that---my own extended standard for
robots.txt is seven years old already, and the original robots.txt standard
was a few years old *then*.
You can read my proposed extentions at
http://www.conman.org/people/spc/robots2.html
> Maybe something along the lines of the following additions: -
>
> Version: [number]
> 1 - original robots.txt specification syntax
> 2 - future extended robots.txt specification
> blank - falls back to version
That's in my proposed extentions.
> Sugested incase for example Version 2 extends the
> already
> existing Allow: line
Yup.
> Interval: [number] [interval]
> [number] - numerical value
> [interval] - h[ours], d[ays], w[eeks], m[onths],
> and y[ears].
>
> As suggested by Fred Atkinson
Yup. Had both a Visit-time: and Request-Rate: headers.
> AllowTypes: [mimetypes]
Didn't have this, as the last time the topic of extending robots.txt came
up, this wasn't an issue (back in 1996).
And given that my proposed extentions have been out there for seven years,
and I think only Allow: may have been adopted, I don't forsee any other
extentions being adopted anytime soon (and yes, I get the occasional email
about it---maybe one every year or year and a half).
-spc (I could update it again, and clear up some ambiguities left, if
anyone cares ... )
_______________________________________________
Robots mailing list
[EMAIL PROTECTED]
http://www.mccmedia.com/mailman/listinfo/robots
