[
https://issues.apache.org/jira/browse/CRUNCH-481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14309880#comment-14309880
]
Ryan Brush commented on CRUNCH-481:
-----------------------------------
Hey Tom,
You're right that the output committer is constructed with a task context using
a different Job instance. I handled this in the patch by "decorating" the job
instance passed to the task context as well. (And it did indeed fail before I
handled that additional Job instance, as one would expect with your comment.)
So I think the logic here is okay. In any case, the tests for Kite, Crunch, and
our local usage all pass. I can't say this is the most elegant code I've ever
written, but these hoops seem to be necessary to simulate multiple output
formats in a job.
Thanks for the clarification on the committer logic in Hadoop 1. That explains
what I saw.
> Support independent output committers for multiple outputs
> ----------------------------------------------------------
>
> Key: CRUNCH-481
> URL: https://issues.apache.org/jira/browse/CRUNCH-481
> Project: Crunch
> Issue Type: Bug
> Components: Core
> Reporter: Aniket Kulkarni
> Assignee: Josh Wills
> Priority: Minor
> Fix For: 0.12.0
>
> Attachments: CRUNCH-481-hadoop-2-compat.patch, CRUNCH-481.patch,
> CRUNCH-481.patch, CRUNCH-481.patch, CRUNCH-481c.patch
>
>
> I faced this issue while trying to write to Kite and HDFS in the same
> pipeline. A similar issue was logged for Kite[1][2].
> I was attempting to write a PCollection to Kite and a different PTable to
> HDFS as a text file. The write to Kite succeeded, however the write to HDFS
> only produced a _SUCCESS file with no text file.
> Commenting out the write to Kite seems to solve the issue and I can see the
> text file being written.
> [1] - https://issues.cloudera.org/browse/CDK-756
> [2] -
> http://mail-archives.apache.org/mod_mbox/crunch-dev/201401.mbox/%3ccaf-wd4qcue0toh3qewpdnnom3u786pvjlgh7t6go_abctpl...@mail.gmail.com%3E
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
