[
https://issues.apache.org/jira/browse/NUTCH-966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sebastian Nagel updated NUTCH-966:
----------------------------------
Fix Version/s: 1.8
> Behavior of NOINDEX,FOLLOW is not intuitive
> -------------------------------------------
>
> Key: NUTCH-966
> URL: https://issues.apache.org/jira/browse/NUTCH-966
> Project: Nutch
> Issue Type: Improvement
> Components: indexer, parser
> Affects Versions: 1.2
> Reporter: Josh Pavel
> Priority: Minor
> Fix For: 2.3, 1.8
>
>
> If a page has NOINDEX,FOLLOW for the ROBOTS metatag, Nutch will still create
> a document that can be found in the index via metatag or URL matching.
> Instead, Nutch should rely on doc or parse metadata but nothing should be
> stored by the html parser. (thanks to Julien Nioche for helping me to
> understand the issue).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
