[
https://issues.apache.org/jira/browse/NUTCH-966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lewis John McGibbney updated NUTCH-966:
---------------------------------------
Fix Version/s: 2.2
1.7
> Behavior of NOINDEX,FOLLOW is not intuitive
> -------------------------------------------
>
> Key: NUTCH-966
> URL: https://issues.apache.org/jira/browse/NUTCH-966
> Project: Nutch
> Issue Type: Improvement
> Components: indexer, parser
> Affects Versions: 1.2
> Reporter: Josh Pavel
> Priority: Minor
> Fix For: 1.7, 2.2
>
>
> If a page has NOINDEX,FOLLOW for the ROBOTS metatag, Nutch will still create
> a document that can be found in the index via metatag or URL matching.
> Instead, Nutch should rely on doc or parse metadata but nothing should be
> stored by the html parser. (thanks to Julien Nioche for helping me to
> understand the issue).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
