[
https://issues.apache.org/jira/browse/NUTCH-1074?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13098820#comment-13098820
]
Markus Jelsma edited comment on NUTCH-1074 at 9/7/11 9:56 AM:
--------------------------------------------------------------
Finally got some numbers to share from a running test:
maxNumSegments = 3
topN = 250.000
Selector reduce output records = 750.000
happen
The above looks fine. The generator selects exactly numSegments * topN records
to be consumed by the following numSegments partitioners. Here's the number of
output reducer records of the following three partitioned segments in order:
1: 471.428
2: 171.562
3: 107.010
The strange thing is that the number of reduce output records exactly matches
the total number of map input records. This is not what i had expected. The
generator partitions by host and has a limit on the number of hosts per queue
so i would expect each segment to contain slightly less records than topN but
certainly not more than topN.
What should we exactly expect?
was (Author: markus17):
Finally got some numbers to share from a running test:
maxNumSegments = 3
topN = 250.000
Selector reduce output records = 750.000
happen
The above looks fine. The generator selects exactly numSegments * topN records
to be consumed by the following numSegments partitioners. Here's the number of
output reducer records of the following three partitioned segments in order:
1: 471.428
2: 171.562
3: 107.010
The strange thing is that the number of reduce output records exactly matches
the total number of map input records. This is not what i had expected. The
generator partitions by host and has a limit on the number of hosts per queue
so i would expect each segment to contain slightly less records than topN.
What should we exactly expect?
> topN is ignored with maxNumSegments
> -----------------------------------
>
> Key: NUTCH-1074
> URL: https://issues.apache.org/jira/browse/NUTCH-1074
> Project: Nutch
> Issue Type: Bug
> Components: generator
> Affects Versions: 1.3
> Reporter: Markus Jelsma
> Fix For: 1.4
>
>
> When generating segments with topN and maxNumSegments, topN is not respected.
> It looks like the first generated segment contains topN * maxNumSegments of
> URLs's, at least the number of map input records roughly matches.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
