[
https://issues.apache.org/jira/browse/PIG-4703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14959700#comment-14959700
]
Rohini Palaniswamy commented on PIG-4703:
-----------------------------------------
[~daijy],
Can we actually get rid of stores variable in TezOperator instead of making
it transient and also get rid of TezCompilerUtil.isIntermediateReducer method
and inline it into TezOperator.isIntermediateReducer method. Can get rid of the
setter as well. That will clean up some code.
> TezOperator.stores shall not ship to backend
> --------------------------------------------
>
> Key: PIG-4703
> URL: https://issues.apache.org/jira/browse/PIG-4703
> Project: Pig
> Issue Type: Bug
> Components: tez
> Reporter: Daniel Dai
> Assignee: Daniel Dai
> Fix For: 0.16.0, 0.15.1
>
> Attachments: PIG-4703-1.patch
>
>
> We end up of shipping all physical plan to the backend. One direct result is
> UDFContext is not set in OutputCommitter. Here is how:
> 1. PigGraceShuffleVertexManager deserialize "pig.tez.plan".
> TezOperator.stores reference physical plan so physical plan get deserialize.
> 2. When deserialize POUserFunc, POUserFunc.instantiateFunc is invoked. Inside
> it, POUserFunc.setFuncInputSchema would use UDFContext and leave an entry
> there (though it is empty map).
> 3. MapRedUtil.setupUDFContext would only deserialize if UDFContext is empty,
> since we already touch UDFContext with some maps, UDFContext is not
> deserialized
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
