[
https://issues.apache.org/jira/browse/NUTCH-710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13528748#comment-13528748
]
zm commented on NUTCH-710:
--------------------------
I have implemented non-canonical page detection by means of modifying
parse-html plugin. Though I am not that familiar with Nutch architecture and
not sure if my implementation is in line with it. What I did is to have utility
method boolean isCanonical(Node root, String baseUrl) which returns status of
currently parsed html page: true if it is proper page, false if it is non
canonical. Then in parse-html plugin's HtmlParser.getParse I call isCanonical
and return from the method with ParseStatus
if(!utils.isCanonical(root, baseUrl)
return ParseStatusUtils.getEmptyParse(9999, "Non canonical page", getConf());
Is this the right way to do it (of cause this needs to be made configurable)?
If someone more familiar with Nutch would confirm or suggest more proper way
I'd submit a patch.
> Support for rel="canonical" attribute
> -------------------------------------
>
> Key: NUTCH-710
> URL: https://issues.apache.org/jira/browse/NUTCH-710
> Project: Nutch
> Issue Type: New Feature
> Affects Versions: 1.1
> Reporter: Frank McCown
> Priority: Minor
> Fix For: 1.7
>
>
> There is a the new rel="canonical" attribute which is
> now being supported by Google, Yahoo, and Live:
> http://googlewebmastercentral.blogspot.com/2009/02/specify-your-canonical.html
> Adding support for this attribute value will potentially reduce the number of
> URLs crawled and indexed and reduce duplicate page content.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
