[
https://issues.apache.org/jira/browse/NUTCH-1339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14642462#comment-14642462
]
Markus Jelsma commented on NUTCH-1339:
--------------------------------------
+1
> Default URL normalization rules to remove page anchors completely
> -----------------------------------------------------------------
>
> Key: NUTCH-1339
> URL: https://issues.apache.org/jira/browse/NUTCH-1339
> Project: Nutch
> Issue Type: Bug
> Affects Versions: nutchgora, 1.6
> Reporter: Sebastian Nagel
> Fix For: 1.11
>
> Attachments: NUTCH-1339-2.patch, NUTCH-1339.patch
>
>
> The default rules of URLNormalizerRegex remove the anchor up to the first
> occurrence of ? or &. The remaining part of the anchor is kept
> which may cause a large, possibly infinite number of outlinks when the same
> document
> fetched again and again with different URLs,
> see http://www.mail-archive.com/user%40nutch.apache.org/msg05940.html
> Parameters in inner-page anchors are a common practice in AJAX web sites.
> Currently, crawling AJAX content is not supported (NUTCH-1323).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
