[
https://issues.apache.org/jira/browse/NUTCH-1113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13866635#comment-13866635
]
Markus Jelsma commented on NUTCH-1113:
--------------------------------------
Alright, NUTCH-1113 isn't correct as well. Overall it looks like NUTCH-1113
doesn't index some documents because they came from redirects (LINKED) and
NUTCH-1616 sometimes incorrectly indexes documents that are already removed.
Indexing stats for NUTCH-1113:
{code}
14/01/09 12:24:22 INFO mapred.JobClient: Deleted redirects=264
14/01/09 12:24:22 INFO mapred.JobClient: Added documents=1112509
14/01/09 12:24:22 INFO mapred.JobClient: Deleted documents=147
14/01/09 12:24:22 INFO mapred.JobClient: Deleted robots noindex=534315
{code}
Indexing stats for NUTCH-1616:
{code}
14/01/09 12:38:29 INFO mapred.JobClient: Deleted redirects=143
14/01/09 12:38:29 INFO mapred.JobClient: Added documents=1113595
14/01/09 12:38:29 INFO mapred.JobClient: Deleted documents=147
14/01/09 12:38:29 INFO mapred.JobClient: Deleted robots noindex=534315
{code}
Consider example.org/home, redirected to from www.example.org. This documents
MUST be indexed:
NUTCH-1113:
{code}
Crawl Fetch::
Version: 7
Status: 67 (linked)
Fetch time: Thu Jan 02 23:48:00 UTC 2014
Modified time: Thu Jan 01 00:00:00 UTC 1970
Retries since fetch: 0
Retry interval: 604800 seconds (7 days)
Score: 0.0
Signature: null
Metadata:
_ngt_=1388706360785
Content-Type=text/html
_pst_=temp_moved(13), lastModified=0: http://www.example.org/home/
_repr_=http://www.example.org/home/
{code}
NUTCH-1616:
{code}
Crawl Fetch::
Version: 7
Status: 33 (fetch_success)
Fetch time: Tue Dec 31 00:32:21 UTC 2013
Modified time: Thu Jul 11 14:33:40 UTC 2013
Retries since fetch: 0
Retry interval: 3060195 seconds (35 days)
Score: 0.0
Signature: 22568059a6934d5ef3f7bf9c8537530f
Metadata:
_ngt_=1388449783232
Content-Type=text/html
_pst_=success(1), lastModified=0
{code}
In this case NUTCH-1616 is correct.
Consider another URL that MUST be indexed:
NUTCH-1113:
{code}
Crawl Fetch::
Version: 7
Status: 33 (fetch_success)
Fetch time: Sat Jan 04 22:50:43 UTC 2014
Modified time: Wed Dec 18 10:33:26 UTC 2013
Retries since fetch: 0
Retry interval: 907200 seconds (10 days)
Score: 0.0
Signature: 5bb8f8efb11fc9b66f65021ced686318
Metadata:
_ngt_=1388875577470
Content-Type=application/xhtml+xml
_pst_=success(1), lastModified=0
{code}
NUTCH-1616:
{code}
Crawl Fetch::
Version: 7
Status: 35 (fetch_redir_temp)
Fetch time: Wed Dec 25 10:43:59 UTC 2013
Modified time: Wed Dec 18 10:33:26 UTC 2013
Retries since fetch: 0
Retry interval: 604800 seconds (7 days)
Score: 0.0
Signature: 5bb8f8efb11fc9b66f65021ced686318
Metadata:
_ngt_=1387967968581
Content-Type=text/html
_pst_=temp_moved(13), lastModified=0:
http://www...ijn-lipsy/?doing_wp_cron=1387968209.7064809799194335937500
{code}
NUTCH-1113 is correct.
> Merging segments causes URLs to vanish from crawldb/index?
> ----------------------------------------------------------
>
> Key: NUTCH-1113
> URL: https://issues.apache.org/jira/browse/NUTCH-1113
> Project: Nutch
> Issue Type: Bug
> Affects Versions: 1.3
> Reporter: Edward Drapkin
> Fix For: 1.9
>
> Attachments: NUTCH-1113-trunk.patch, merged_segment_output.txt,
> unmerged_segment_output.txt
>
>
> When I run Nutch, I use the following steps:
> nutch inject crawldb/ url.txt
> repeated 3 times:
> nutch generate crawldb/ segments/ -normalize
> nutch fetch `ls -d segments/* | tail -1`
> nutch parse `ls -d segments/* | tail -1`
> nutch update crawldb `ls -d segments/* | tail -1`
> nutch mergesegs merged/ -dir segments/
> nutch invertlinks linkdb/ -dir merged/
> nutch index index/ crawldb/ linkdb/ -dir merged/ (I forward ported the lucene
> indexing code from Nutch 1.1).
> When I crawl with merging segments, I lose about 20% of the URLs that wind up
> in the index vs. when I crawl without merging the segments. Somehow the
> segment merger causes me to lose ~20% of my crawl database!
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
