[
https://issues.apache.org/jira/browse/NUTCH-2652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16657954#comment-16657954
]
Hudson commented on NUTCH-2652:
-------------------------------
FAILURE: Integrated in Jenkins build Nutch-trunk #3568 (See
[https://builds.apache.org/job/Nutch-trunk/3568/])
NUTCH-2652 Fetcher launches more fetch tasks than fetch lists - properly
(snagel:
[https://github.com/apache/nutch/commit/a6de472556c0bf3e38006999a3b5eb8ad67020b6])
* (edit) src/java/org/apache/nutch/fetcher/Fetcher.java
> Fetcher launches more fetch tasks than fetch lists
> --------------------------------------------------
>
> Key: NUTCH-2652
> URL: https://issues.apache.org/jira/browse/NUTCH-2652
> Project: Nutch
> Issue Type: Bug
> Components: fetcher
> Affects Versions: 1.15
> Environment: Hadoop, distributed mode (cluster of 22 nodes), CDH
> 5.15.1, Nutch built on recent master.
> Seen the first time right now, although running since two months with Nutch
> 1.15. But the constraints causing inputs to be split may change from run to
> run.
> Reporter: Sebastian Nagel
> Assignee: Sebastian Nagel
> Priority: Critical
> Fix For: 1.16
>
>
> Fetcher may launch more fetcher tasks than there are fetch lists:
> {noformat}
> 18/10/15 07:27:26 INFO input.FileInputFormat: Total input paths to process :
> 128
> 18/10/15 07:27:26 INFO mapreduce.JobSubmitter: number of splits:187
> {noformat}
> That's one design principle of Nutch as a MapRecude-based crawler: to ensure
> politeness and a guaranteed delay between requests to the same host/domain/ip
> all items of one host/domain/ip are put by Generator into the same fetch
> list. A fetch list may not be split because that would violate the politeness
> constraints - multiple fetcher tasks processing the splits of one fetch list
> then may send requests to the same host/domain/ip in parallel. See [~ab]'s
> chapter about Nutch in [Hadoop the definitive guide (3rd
> edition)|https://www.safaribooksonline.com/library/view/hadoop-the-definitive/9781449328917/ch16.html#NutchFetcher].
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
