[
https://issues.apache.org/jira/browse/PIG-4202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14151964#comment-14151964
]
Bikas Saha commented on PIG-4202:
---------------------------------
To be clear, none of my above comment was for processors. It was for the
OutputCommitter API in Tez which runs in the AM per vertex per output. The
lifecycle is
committer.initialize()->committer->setupOutputs()->committer->commitOutput().
In the absence of side effects commitOutput() is the equivalent of close(). We
can add a new committer.close() which would be called immediately after
committer.commitOutput(). So its value is not clear. Does PigOutputCommitter
derive from OutputCommitter in Tez?
Questions to understand this better. Are there any issues in cleaning up within
commitOutput() after commit operations have completed? Do multiple committers
in a vertex (vertex with multiple HDFS outputs) or multiple outputs in
different vertices work today? If yes, then why are the statics not causing a
problem?
> UDFContext.isFrontend return true for OutputCommitter in local mode
> -------------------------------------------------------------------
>
> Key: PIG-4202
> URL: https://issues.apache.org/jira/browse/PIG-4202
> Project: Pig
> Issue Type: Bug
> Components: impl
> Reporter: Daniel Dai
> Assignee: Daniel Dai
> Fix For: 0.14.0
>
> Attachments: PIG-4202-1.patch, PIG-4202-2.patch
>
>
> We switch to use UDFContext.isFrontend in PIG-4187. However, it does not work
> as expected in local mode. This causes some local mode test fail
> (TestOrcStoragePushdown.createInputData).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
