[
https://issues.apache.org/jira/browse/CRUNCH-673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16614861#comment-16614861
]
Gabriel Reid commented on CRUNCH-673:
-------------------------------------
Uploaded an alternate patch ( [^CRUNCH-673.2.patch] ) that takes a different
approach: instead of reworking things in o.a.c.lib.Sort, this one just relaxes
the preconditions in TotalOrderPartitioner to only check that the number of
partitions is equal to or smaller than the number of configured reducers.
Although this approach has a slight risk (i.e. could hide other problems), it
has non impact on the actual execution of the pipeline. The issue with the
original patch is that the execution of the pipeline is broken up by the need
to materialize the reservoir-sampled data to verify that it contains enough
elements.
I've left the integration test in this new patch as well. Preliminary testing
of this patch in our specific use case seems to show that it is indeed working
as needed.
> Sort fails when using more reducers than records
> ------------------------------------------------
>
> Key: CRUNCH-673
> URL: https://issues.apache.org/jira/browse/CRUNCH-673
> Project: Crunch
> Issue Type: Bug
> Reporter: Gabriel Reid
> Priority: Minor
> Attachments: CRUNCH-673.2.patch, CRUNCH-673.patch
>
>
> We've run into an issue where running Sort with a number of reducers that is
> higher than the number of records to be sorted fails.
> The way in which this occurs is that a large PCollection is filtered down to
> almost nothing (say 10 records), and that filtered PCollection is passed in
> to Sort. Sort configures n reducers for the small PCollection (because it
> doesn't realize that it has been filtered so aggressively), so then there are
> for example 20 reducers configured. Reservoir sampling is used to build up
> the partition definitions for the TotalOrderPartitioner, but because there
> are only 10 records in the filtered PCollection, only 10 partitions are
> defined for the TotalOrderPartitioner. This then causes a precondition in
> TotalOrderPartitioner to fail, because the number of partitions in the
> partitions file doesn't match up with the number of configured reducers.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
