[
https://issues.apache.org/jira/browse/NUTCH-1616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13718296#comment-13718296
]
Markus Jelsma commented on NUTCH-1616:
--------------------------------------
Well spotted!! I'm positive i did it right because i actually use a script for
that which has it enabled as well. I manually retyped it to be a oneline for
this comment box instead of copying a script:) Sorry for the needless confusion.
About the fetchTime thing, i'm sure that now i think about that the current
IndexerMapReduce can produce bad results already because it might skip/delete
records if multiple segments are input. It checks if the fetchDatum is a
redirect, not modified or gone for the first incoming fetchDatum! I'll open an
issue for that!
Thanks for that pointer!
> SegmentMerger missing proper crawl_fetch datum
> ----------------------------------------------
>
> Key: NUTCH-1616
> URL: https://issues.apache.org/jira/browse/NUTCH-1616
> Project: Nutch
> Issue Type: Bug
> Affects Versions: 1.7
> Reporter: Markus Jelsma
> Assignee: Markus Jelsma
> Priority: Critical
> Fix For: 1.8
>
>
> Merged 26036 vs. unmerged 26038 indexed documents! There are two records on
> the merged segment that no longer have a crawl_fetch CrawlDatum with a
> fetch_success status. Instead, the only crawl_fetch CrawlDatum has status
> linked!
> The original segment two crawl_fetch CrawlDatums with linked and the
> fetch_success status.
> Without the fetch_success of not_modified status it is not going to be
> indexed.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
