[
https://issues.apache.org/jira/browse/NUTCH-1113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13915831#comment-13915831
]
Sebastian Nagel commented on NUTCH-1113:
----------------------------------------
Results of tests: The number of documents in index after all segments have been
* (A) indexed in chronological order segment by segment or
* (B) merged into one segment which then has been indexed
is shown below. B has been run twice: (1) without any patch, and (2) using the
patch as of 2014-02-21. IndexerMapReduce was patched by
NUTCH-1706-trunk-v2.patch for all 3 runs.
|| || coll 1 || coll 2 || coll 3 ||
| A seg-by-seg | 22178 | 6959 | 45944 |
| B1 merged | 21122 | 6579 | 46029 |
| B2 patched, merged | 22161 | 6959 | 46135 |
3 collections have been tested, all of them with ~100 segments and >100.000
URLs, but with many redirects, robots noindex etc. (far more than indexable
documents). With patch (B2 compared to B1) the index contains more documents.
For collection 2 it's now equal to the expected number (A). For the other two
collections the numbers differ but that's because of problems in
IndexerMapReduce (NUTCH-1708 and NUTCH-1732).
+1 to commit [~markus17]'s latest patch.
> Merging segments causes URLs to vanish from crawldb/index?
> ----------------------------------------------------------
>
> Key: NUTCH-1113
> URL: https://issues.apache.org/jira/browse/NUTCH-1113
> Project: Nutch
> Issue Type: Bug
> Affects Versions: 1.3
> Reporter: Edward Drapkin
> Priority: Blocker
> Fix For: 1.9
>
> Attachments: NUTCH-1113-junit.patch, NUTCH-1113-junit.patch,
> NUTCH-1113-junit.patch, NUTCH-1113-junit.patch, NUTCH-1113-junit.patch,
> NUTCH-1113-junit.patch, NUTCH-1113-trunk.patch, NUTCH-1113-trunk.patch,
> merged_segment_output.txt, unmerged_segment_output.txt
>
>
> When I run Nutch, I use the following steps:
> nutch inject crawldb/ url.txt
> repeated 3 times:
> nutch generate crawldb/ segments/ -normalize
> nutch fetch `ls -d segments/* | tail -1`
> nutch parse `ls -d segments/* | tail -1`
> nutch update crawldb `ls -d segments/* | tail -1`
> nutch mergesegs merged/ -dir segments/
> nutch invertlinks linkdb/ -dir merged/
> nutch index index/ crawldb/ linkdb/ -dir merged/ (I forward ported the lucene
> indexing code from Nutch 1.1).
> When I crawl with merging segments, I lose about 20% of the URLs that wind up
> in the index vs. when I crawl without merging the segments. Somehow the
> segment merger causes me to lose ~20% of my crawl database!
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
