[
https://issues.apache.org/jira/browse/CRUNCH-670?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Ortiz updated CRUNCH-670:
-------------------------------
Attachment: CRUNCH-670-pt2.patch
> Make the AvroPathPerKeyTarget work with the SparkRuntime
> --------------------------------------------------------
>
> Key: CRUNCH-670
> URL: https://issues.apache.org/jira/browse/CRUNCH-670
> Project: Crunch
> Issue Type: Improvement
> Reporter: Josh Wills
> Assignee: Josh Wills
> Priority: Major
> Attachments: CRUNCH-670-pt2.patch, CRUNCH-670.patch
>
>
> There is an issue where the AvroPathPerKeyTarget won't properly copy the
> output of a Spark pipeline from the temp directory to the target directory
> because it assumes it will always get a valid Crunch output index (0, 1, 2,
> ...) and the SparkRuntime passes -1 to the Target's output handler method (to
> signal that it's the only output for the job.) I _think_ the right move is to
> have AvroPathPerKeyTarget rewrite a -1 index to 0 so as not to break any
> other implementations that depend on the SparkRuntime's behavior.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
