[
https://issues.apache.org/jira/browse/PIG-4458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14381033#comment-14381033
]
Rohini Palaniswamy commented on PIG-4458:
-----------------------------------------
Both. My box and our internal Jenkins build. Pig unit tests take 6+ hours to
run. So there is no Apache build that runs all the tests on a patch. There used
to be one build that runs a subset of tests (test/commit-tests) , but that is
broken and needs to be fixed. Anyways TestMergeJoin is not part of that.
> Support UDFs in a FOREACH Before a Merge Join
> ---------------------------------------------
>
> Key: PIG-4458
> URL: https://issues.apache.org/jira/browse/PIG-4458
> Project: Pig
> Issue Type: New Feature
> Reporter: William Watson
> Assignee: William Watson
> Fix For: 0.15.0
>
> Attachments: PIG-4458-FixTestFailure.patch,
> PIG-4458.04.remove-merge-join-udf-restriction.patch,
> PIG-4458.05.remove-merge-join-udf-restriction.patch
>
>
> Right now, the MapSideMergeValidator outright rejects any foreach that has a
> UDF in it:
> {code}
> private boolean isAcceptableForEachOp(Operator lo) throws
> LogicalToPhysicalTranslatorException {
> if (lo instanceof LOForEach) {
> OperatorPlan innerPlan = ((LOForEach) lo).getInnerPlan();
> validateMapSideMerge(innerPlan.getSinks(), innerPlan);
> return !containsUDFs((LOForEach) lo);
> } else {
> return false;
> }
> }
> {code}
> There is a TODO for this later on in that same class (inside containsUDFs):
> {code}
> // TODO (dvryaboy): in the future we could relax this rule by tracing what
> fields
> // are being passed into the UDF, and only refusing if the UDF is working on
> the
> // join key. Transforms of other fields should be ok.
> {code}
> We should do the TODO and relax this requirement or just remove it altogether
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
