[
https://issues.apache.org/jira/browse/CRUNCH-241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Josh Wills updated CRUNCH-241:
------------------------------
Attachment: CRUNCH-241.patch
The patch. Learned this trick from working with [~srowen].
> Write side outputs from the Mapper stage of a MapReduce job
> -----------------------------------------------------------
>
> Key: CRUNCH-241
> URL: https://issues.apache.org/jira/browse/CRUNCH-241
> Project: Crunch
> Issue Type: Bug
> Components: Core
> Reporter: Josh Wills
> Assignee: Josh Wills
> Attachments: CRUNCH-241.patch
>
>
> Right now, Crunch always writes output files from the "last" stage of
> whatever kind of job it runs: either the reduce-side of a MapReduce job, or
> the map-side of a map-only job. This often leads to situations where we have
> to re-process the same input twice, once for the map-side outputs and again
> for the reduce-side outputs.
> This change adds the ability for Crunch to write side outputs from the mapper
> phase of a MapReduce job (i.e., we can write output Targets from both the map
> side and the reduce side.) This should help lots of pipelines that implement
> these types of writes execute much faster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
