[ http://issues.apache.org/jira/browse/NUTCH-136?page=all ]
Andrzej Bialecki closed NUTCH-136:
-----------------------------------
Resolution: Duplicate
Thank you for investigating this. I'm closing this issue, further discussion
should follow to NUTCH-186.
> mapreduce segment generator generates 50 % less than excepted urls
> --------------------------------------------------------------------
>
> Key: NUTCH-136
> URL: http://issues.apache.org/jira/browse/NUTCH-136
> Project: Nutch
> Type: Bug
> Versions: 0.8-dev
> Reporter: Stefan Groschupf
> Priority: Critical
>
> We notice that segments generated with the map reduce segment generator
> contains only 50 % of the expected urls. We had a crawldb with 40 000 urls
> and the generate commands only created a 20 000 pages segment. This also
> happened with the topN parameter, we everytime got around 50 % of the
> expected urls.
> I tested the PartitionUrlByHost and it looks like it does its work. However
> we fixed the problem by changing two things:
> First we set the partition to a normal hashPartitioner.
> Second we changed Generator.java line 48:
> limit = job.getLong("crawl.topN",Long.MAX_VALUE)/job.getNumReduceTasks();
> to:
> limit = job.getLong("crawl.topN",Long.MAX_VALUE);
> Now it works as expected.
> Has anyone a idea what the real source of this problem can be?
> In general this is bug has the effect that all map reduce users fetch only 50
> % of it's urls per iteration.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
