[
https://issues.apache.org/jira/browse/NUTCH-2219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15157091#comment-15157091
]
Hudson commented on NUTCH-2219:
-------------------------------
SUCCESS: Integrated in Nutch-trunk #3350 (See
[https://builds.apache.org/job/Nutch-trunk/3350/])
NUTCH-2219 Criteria order to be configurable in DeduplicationJob (markus:
[http://svn.apache.org/viewvc/nutch/trunk/?view=rev&rev=1731651])
* trunk/CHANGES.txt
* trunk/src/java/org/apache/nutch/crawl/DeduplicationJob.java
> Criteria order to be configurable in DeduplicationJob
> -----------------------------------------------------
>
> Key: NUTCH-2219
> URL: https://issues.apache.org/jira/browse/NUTCH-2219
> Project: Nutch
> Issue Type: New Feature
> Components: crawldb
> Affects Versions: 1.11
> Reporter: Ron van der Vegt
> Assignee: Markus Jelsma
> Fix For: 1.12
>
> Attachments: NUTCH-2219.patch, NUTCH-2219.patch
>
>
> Current implementation:
> "This command takes a path to a crawldb as parameter and finds duplicates
> based on the signature. If several entries share the same signature, the one
> with the highest score is kept. If the scores are the same, then the fetch
> time is used to determine which one to keep with the most recent one being
> kept. If their fetch times are the same we keep the one with the shortest
> URL."
> The order in which the main document is selected is currently not changeable.
> Therefore I think this option would be nice:
> -compareOrder <score>,<fetchTime>,<urlLength>
> I have written a patch on trunk (rev 1730516). I'm looking forward for any
> peer review.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
