Author: theli Date: 2006-03-14 09:54:44 +0100 (Tue, 14 Mar 2006) New Revision: 1886
Modified: trunk/source/de/anomic/data/robotsParser.java Log: testcommit with subversive Modified: trunk/source/de/anomic/data/robotsParser.java =================================================================== --- trunk/source/de/anomic/data/robotsParser.java 2006-03-13 23:23:45 UTC (rev 1885) +++ trunk/source/de/anomic/data/robotsParser.java 2006-03-14 08:54:44 UTC (rev 1886) @@ -65,13 +65,20 @@ /* * A class for Parsing robots.txt files. * It only parses the Deny Part, yet. - * * + * + * Robots RFC * http://www.robotstxt.org/wc/norobots-rfc.html * * TODO: * - On the request attempt resulted in temporary failure a robot * should defer visits to the site until such time as the resource * can be retrieved. + * + * - Extended Standard for Robot Exclusion + * See: http://www.conman.org/people/spc/robots2.html + * + * - Robot Exclusion Standard Revisited + * See: http://www.kollar.com/robots.html */ public final class robotsParser{ _______________________________________________ YaCy-svn mailing list [email protected] http://lists.berlios.de/mailman/listinfo/yacy-svn
