[
https://issues.apache.org/jira/browse/HADOOP-3019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Douglas updated HADOOP-3019:
----------------------------------
Attachment: 3019-3.patch
Updated based on Owen's feedback:
* Changed RandomSampler to sample evenly across all splits, rather than evenly
from each split
* Used double instead of float for sampling rate
This patch also modifies the sort example to demonstrate how to use
InputSampler in a job. This requires examples to depend on tools.
> want input sampler & sorted partitioner
> ---------------------------------------
>
> Key: HADOOP-3019
> URL: https://issues.apache.org/jira/browse/HADOOP-3019
> Project: Hadoop Core
> Issue Type: New Feature
> Components: mapred
> Reporter: Doug Cutting
> Assignee: Chris Douglas
> Fix For: 0.19.0
>
> Attachments: 3019-0.patch, 3019-1.patch, 3019-2.patch, 3019-3.patch
>
>
> The input sampler should generate a small, random sample of the input, saved
> to a file.
> The partitioner should read the sample file and partition keys into
> relatively even-sized key-ranges, where the partition numbers correspond to
> key order.
> Note that when the sampler is used for partitioning, the number of samples
> required is proportional to the number of reduce partitions. 10x the
> intended reducer count should give good results.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
