[
https://issues.apache.org/jira/browse/NUTCH-2219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Markus Jelsma updated NUTCH-2219:
---------------------------------
Attachment: NUTCH-2219.patch
Thanks, looks fine!
Slightly updated patch:
* changed usage output
* changed url to urlLength for clarity
Will commit sometime soon unless objections
> Dedup script, allow users to change the order in which main documents are
> selected.
> -----------------------------------------------------------------------------------
>
> Key: NUTCH-2219
> URL: https://issues.apache.org/jira/browse/NUTCH-2219
> Project: Nutch
> Issue Type: New Feature
> Components: crawldb
> Reporter: Ron van der Vegt
> Assignee: Markus Jelsma
> Attachments: NUTCH-2219.patch, NUTCH-2219.patch
>
>
> Current implementation:
> "This command takes a path to a crawldb as parameter and finds duplicates
> based on the signature. If several entries share the same signature, the one
> with the highest score is kept. If the scores are the same, then the fetch
> time is used to determine which one to keep with the most recent one being
> kept. If their fetch times are the same we keep the one with the shortest
> URL."
> The order in which the main document is selected is currently not changeable.
> Therefore I think this option would be nice:
> -compareOrder <score>,<fetchTime>,<url>
> I have written a patch on trunk (rev 1730516). I'm looking forward for any
> peer review.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
