[
https://issues.apache.org/jira/browse/NUTCH-1616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13716248#comment-13716248
]
Markus Jelsma commented on NUTCH-1616:
--------------------------------------
Alright, i thought i fixed it with a modification very similar to NUTCH-1113 -
it needs ||, not && - but i'm not so sure anymore what i should consider the
correct behaviour. I test my changes by indexing the data, usually i either
index a merged segment or a directory of unmerged segments but there is always
a difference if i index ~100.000 documents. I also decided to index all those
segments with individual jobs and the outcome is also different.
For example:
# 106678 documents with nutch solrindex crawl/crawldb merged_segment/
# 106669 documents with nutch solrindex crawl/crawldb -dir unmerged_segments/
# 106694 documents with for file in `hadoop fs -ls
sitesearch/segments/unfetched | grep segments | cut -d / -f 7` ; nutch
solrindex crawl/crawldb unmerged_segment/; done
Well, this is crazy! Since all indexing actions are incremental i would assume
the last is correct. But it seems that indexing a complete directory never
worked at all and that reindexing the merged segments with a patch yields
better results.
I haven't found out yet but i'd first share those results here.
Cheers
> SegmentMerger missing proper crawl_fetch datum
> ----------------------------------------------
>
> Key: NUTCH-1616
> URL: https://issues.apache.org/jira/browse/NUTCH-1616
> Project: Nutch
> Issue Type: Bug
> Affects Versions: 1.7
> Reporter: Markus Jelsma
> Assignee: Markus Jelsma
> Priority: Critical
> Fix For: 1.8
>
>
> Merged 26036 vs. unmerged 26038 indexed documents! There are two records on
> the merged segment that no longer have a crawl_fetch CrawlDatum with a
> fetch_success status. Instead, the only crawl_fetch CrawlDatum has status
> linked!
> The original segment two crawl_fetch CrawlDatums with linked and the
> fetch_success status.
> Without the fetch_success of not_modified status it is not going to be
> indexed.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
