On May 27, 2008, at 11:42 AM, Vijay Krishnan wrote:
Hi all,
Do you know what file in Nutch parses robots.txt? If there is no
option in Nutch to ignore robots.txt, I would at least like to modify
the source.
Thanks,
Vijay
src/plugin/lib-http/src/java/org/apache/nutch/protocol/http/api/
RobotRulesParser.java
parses robots.txt
src/plugin/parse-html/src/java/org/apache/nutch/parse/html/
HTMLMetaProcessor.java
parses robot rules from html documents.
eric
Eric J. Christeson
<[EMAIL PROTECTED]>
Information Technology Services (701) 231-8693 (Voice)
Room 242C, IACC Building
North Dakota State University, Fargo, ND 58105-5164
Organizations which design systems are constrained to produce designs
which
are copies of the communication structures of these organizations. (For
example, if you have four groups working on a compiler, you'll get a
4-pass compiler) - Conway's Law
