> Are we still around? Haven't heard anything on this list for quite a
> while.
Yes.
Obligatory-Content: My interest here is that I'm writing a proxy-style
browser that does read-ahead crawling with indexing. It's unclear to
me how much it's behaving as a robot and how much it's behaving as a
browser... (Also it's browsing RDF content not HTML content.)
I wonder about extending robots.txt more in the direction of site
policies, using RDF, possibly with a rule language. (That's another
area of interest for me -- working on a web standard rule language.)
Perhaps merging with P3P, although there's a big difference between
P3P being "here's what this site promises" and robots.txt being
"please behave this way." Maybe they can be seen the same way if we
view robots.txt as saying "I promise to be okay with robots accessing
the site like this...".
-- sandro
_______________________________________________
Robots mailing list
[email protected]
http://www.mccmedia.com/mailman/listinfo/robots
