It was thus said that the Great Walter Underwood once stated:
The record starts with one or more User-agent lines, followed by one or more Disallow lines, as detailed below. Unrecognised headers are ignored.
Right there---last line---"Unrecognised headers are ignored."
Good, the the format can be upgraded, within limits.
[skip to robustness principle discussion]
By the same token, it is the robots.txt parser that "accepts" the robots.txt file, so by the robustness principle, you need to ignore directives you don't understand.
Right, but the robustness principle means that sender cannot rely on the receiver doing that. If each end makes allowance for bugs in the other end, the the protocol is much more likely to work in the real world.
-spc (And be thankful Martijn didn't decide to use RFC-822 style
header lines ... )
It wasn't just Martijn, it was a group of people, all actively involved in robots and web sites. I'm sure that someone was aware that they did not need to fit robots.txt lines on an 80-column Hollerith card.
wunder -- Walter Underwood Principal Architect Verity Ultraseek
_______________________________________________ Robots mailing list [EMAIL PROTECTED] http://www.mccmedia.com/mailman/listinfo/robots
