[
https://issues.apache.org/jira/browse/NUTCH-2232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15163138#comment-15163138
]
Hudson commented on NUTCH-2232:
-------------------------------
SUCCESS: Integrated in Nutch-trunk #3354 (See
[https://builds.apache.org/job/Nutch-trunk/3354/])
NUTCH-2232 DeduplicationJob should decode URL's before length is compared
(markus: [http://svn.apache.org/viewvc/nutch/trunk/?view=rev&rev=1732160])
* trunk/CHANGES.txt
* trunk/src/java/org/apache/nutch/crawl/DeduplicationJob.java
> DeduplicationJob should decode URL's before length is compared
> --------------------------------------------------------------
>
> Key: NUTCH-2232
> URL: https://issues.apache.org/jira/browse/NUTCH-2232
> Project: Nutch
> Issue Type: Bug
> Components: crawldb
> Affects Versions: 1.11
> Reporter: Ron van der Vegt
> Assignee: Markus Jelsma
> Fix For: 1.12
>
> Attachments: NUTCH-2232.patch, NUTCH-2232.patch
>
>
> When certain documents have the same signature de deduplication script will
> elect one as duplicate. The urls are stored url encoded in the crawldb. When
> two urls are compared by url length, the urls are not first decoded. This
> could lead to misleading url length.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
