[
https://issues.apache.org/jira/browse/NUTCH-1616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13718156#comment-13718156
]
Markus Jelsma commented on NUTCH-1616:
--------------------------------------
In any case i get confusing results when running on an existing Hadoop 1.3.0
cluster. I did some testing and indexed the same segment dir for several times
and i get different results, i.e. number of indexed records. Something crazy is
going on here, for example:
Yesterday afternoon:
Deleted redirects 0 4,882 4,882
Added documents 0 106,670 106,670
Deleted by filters 0 3 3
Deleted documents 0 694 694
Deleted robots noindex 0 37,630 37,630
...
Reduce input records 0 0 10,057,140
Reduce input groups 0 0 2,455,106
Combine output records 0 0 0
Physical memory (bytes) snapshot 0 0 300,645,380,096
Reduce output records 0 0 149,879
Virtual memory (bytes) snapshot 0 0 1,584,725,905,408
Map output records 0 0 11,299,577
This morning:
Deleted redirects 0 4,882 4,882
Added documents 0 106,666 106,666
Deleted by filters 0 3 3
Deleted documents 0 694 694
Deleted robots noindex 0 37,630 37,630
...
Reduce input records 0 0 10,064,231
Reduce input groups 0 0 2,455,106
Combine output records 0 0 0
Physical memory (bytes) snapshot 0 0 301,525,434,368
Reduce output records 0 0 149,875
Virtual memory (bytes) snapshot 0 0 1,582,281,064,448
Map output records 0 0 11,299,577
Moment ago:
Deleted redirects 0 4,882 4,882
Added documents 0 106,670 106,670
Deleted by filters 0 3 3
Deleted documents 0 694 694
Deleted robots noindex 0 37,630 37,630
...
Reduce input records 0 0 10,048,765
Reduce input groups 0 0 2,455,106
Combine output records 0 0 0
Physical memory (bytes) snapshot 0 0 301,517,443,072
Reduce output records 0 0 149,879
Virtual memory (bytes) snapshot 0 0 1,582,767,226,880
Map output records 0 0 11,299,577
This happens many times. The number of indexed documents always hovers around
106669! You can clearly see that the input is identical (map output records ==
map input records). But depending on the wheather or cosmic radiation it
decides to input more reduce records.
Anyone to confirm this crazyness?
> SegmentMerger missing proper crawl_fetch datum
> ----------------------------------------------
>
> Key: NUTCH-1616
> URL: https://issues.apache.org/jira/browse/NUTCH-1616
> Project: Nutch
> Issue Type: Bug
> Affects Versions: 1.7
> Reporter: Markus Jelsma
> Assignee: Markus Jelsma
> Priority: Critical
> Fix For: 1.8
>
>
> Merged 26036 vs. unmerged 26038 indexed documents! There are two records on
> the merged segment that no longer have a crawl_fetch CrawlDatum with a
> fetch_success status. Instead, the only crawl_fetch CrawlDatum has status
> linked!
> The original segment two crawl_fetch CrawlDatums with linked and the
> fetch_success status.
> Without the fetch_success of not_modified status it is not going to be
> indexed.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
