You're right about the tendency to ignore the touch date, and in a more general sense the tendency for robots.txt to become stale without automation.
As for the Session attribute, I made reference to it in a prior email (albeit without the capitalization): By link funneling I'm referring to links that contain random session identifiers, causing the same pages to be served up perpetually with different anchor tags. Most robots could benefit from a line that identified these types of urls by their session identifier, for example: session=jsessionid. This simple enhancement would benefit both the robot developers and the site developers. The robots would no longer need to identify these urls by manual or automated profiling, they could simply extract the session identifier from links that matched the mask. The site developers would prevent the useless traffic that's presently involved in inferring the random session identifiers. Extending this idea, perhaps the specification could allow robots to substitute their own agent names for the session identifiers. This would allow for a loose type of referral tracking. As a side effect it would also cause robots that spoof their agent names to implicate competing robots. -----Original Message----- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Sean 'Captain Napalm' Conner Sent: Sunday, January 11, 2004 6:05 PM To: [EMAIL PROTECTED] Subject: Re: [Robots] Another approach It was thus said that the Great Matthew Meadows once stated: > > I do like the Interval attribute, that makes perfect sense to me. > There's a lot we could do with the same basic concept. For instance, > we could add a touch date to the file to indicate when the site was > last updated, so that even if the interval has passed robots would not > need Then there is the issue with making sure the robots.txt file is updated with the new timestamp each time the site is updated, and I suspect this step may be ignored for forgotten unless it's automated. > Anybody else interested in the Session attribute? What's the session attribute? -spc (http://www.conman.org/people/spc/robots2.html) _______________________________________________ Robots mailing list [EMAIL PROTECTED] http://www.mccmedia.com/mailman/listinfo/robots _______________________________________________ Robots mailing list [EMAIL PROTECTED] http://www.mccmedia.com/mailman/listinfo/robots
