I know Nutch crawl the website according to Robot protocol if you make that configuration. And it will not fetch and parse the link on the page which contains <meta name="robots" content="nofollow">. But can Nutch process rel-tag likes rel="nofollow" in the tags ...... on the page?
-- View this message in context: http://lucene.472066.n3.nabble.com/Can-Nutch-process-rel-tag-likes-rel-nofollow-tp4001541.html Sent from the Nutch - Dev mailing list archive at Nabble.com.
