[
https://issues.apache.org/jira/browse/PIG-4202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14151916#comment-14151916
]
Bikas Saha commented on PIG-4202:
---------------------------------
To be clear, committer are per vertex in Tez and not per DAG. So there is no
concept of a DAG level committer setup/close. If Pig has static variable issues
then they may show up if/when Pig writes multiple HDFS outputs in different
vertices of the same DAG.
In Tez the OutputCommitter interface, from which all committers are expected to
derive, has an initialize() method that is called upon Vertex initialization.
Could PigOutputCommitter clear its static vars in that method? If not, could
PigOutputCommitter clear its static vars in the setupJob() method (this is an
MR OutputCommitter API). Again, given that committers are per vertex, this
could have implications for multiple vertices writing output that needs commit.
> UDFContext.isFrontend return true for OutputCommitter in local mode
> -------------------------------------------------------------------
>
> Key: PIG-4202
> URL: https://issues.apache.org/jira/browse/PIG-4202
> Project: Pig
> Issue Type: Bug
> Components: impl
> Reporter: Daniel Dai
> Assignee: Daniel Dai
> Fix For: 0.14.0
>
> Attachments: PIG-4202-1.patch, PIG-4202-2.patch
>
>
> We switch to use UDFContext.isFrontend in PIG-4187. However, it does not work
> as expected in local mode. This causes some local mode test fail
> (TestOrcStoragePushdown.createInputData).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
