[
https://issues.apache.org/jira/browse/NUTCH-1113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13915969#comment-13915969
]
Hudson commented on NUTCH-1113:
-------------------------------
FAILURE: Integrated in Nutch-trunk #2545 (See
[https://builds.apache.org/job/Nutch-trunk/2545/])
NUTCH-1113 SegmentMerger can now be safely used to merge segments. If this damn
thing breaks again.... (markus:
http://svn.apache.org/viewvc/nutch/trunk/?view=rev&rev=1572975)
* /nutch/trunk/CHANGES.txt
* /nutch/trunk/src/java/org/apache/nutch/segment/SegmentMerger.java
*
/nutch/trunk/src/test/org/apache/nutch/segment/TestSegmentMergerCrawlDatums.java
> Merging segments causes URLs to vanish from crawldb/index?
> ----------------------------------------------------------
>
> Key: NUTCH-1113
> URL: https://issues.apache.org/jira/browse/NUTCH-1113
> Project: Nutch
> Issue Type: Bug
> Affects Versions: 1.3
> Reporter: Edward Drapkin
> Assignee: Markus Jelsma
> Priority: Blocker
> Fix For: 1.8
>
> Attachments: NUTCH-1113-junit.patch, NUTCH-1113-junit.patch,
> NUTCH-1113-junit.patch, NUTCH-1113-junit.patch, NUTCH-1113-junit.patch,
> NUTCH-1113-junit.patch, NUTCH-1113-trunk-junit-final.patch,
> NUTCH-1113-trunk.patch, NUTCH-1113-trunk.patch, merged_segment_output.txt,
> unmerged_segment_output.txt
>
>
> When I run Nutch, I use the following steps:
> nutch inject crawldb/ url.txt
> repeated 3 times:
> nutch generate crawldb/ segments/ -normalize
> nutch fetch `ls -d segments/* | tail -1`
> nutch parse `ls -d segments/* | tail -1`
> nutch update crawldb `ls -d segments/* | tail -1`
> nutch mergesegs merged/ -dir segments/
> nutch invertlinks linkdb/ -dir merged/
> nutch index index/ crawldb/ linkdb/ -dir merged/ (I forward ported the lucene
> indexing code from Nutch 1.1).
> When I crawl with merging segments, I lose about 20% of the URLs that wind up
> in the index vs. when I crawl without merging the segments. Somehow the
> segment merger causes me to lose ~20% of my crawl database!
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
