[
https://issues.apache.org/jira/browse/PIG-5082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15805050#comment-15805050
]
Rohini Palaniswamy commented on PIG-5082:
-----------------------------------------
[~tmwoodruff],
No. I got that covered. I only need to update the e2e test I had for
PIG-5078 with your case. I will do that shortly. Meanwhile, can you test the
https://issues.apache.org/jira/secure/attachment/12846040/PIG-5078-1.patch and
let me know if that works for you?
> Tez UnionOptimizer creates vertex group with one member
> -------------------------------------------------------
>
> Key: PIG-5082
> URL: https://issues.apache.org/jira/browse/PIG-5082
> Project: Pig
> Issue Type: Bug
> Components: tez
> Affects Versions: 0.16.0
> Reporter: Travis Woodruff
> Assignee: Rohini Palaniswamy
> Priority: Minor
> Fix For: 0.17.0, 0.16.1
>
> Attachments: PIG-5082.patch
>
>
> This script results in a vertex group with one member:
> {code}
> a = LOAD '/tmp/empty.txt' USING PigStorage('\t') AS (x:chararray);
> b = LOAD '/tmp/empty.txt' USING PigStorage('\t') AS (x:chararray);
> c = LOAD '/tmp/empty.txt' USING PigStorage('\t') AS (y:chararray);
> u1 = UNION ONSCHEMA a, b;
> SPLIT u1 INTO r IF x != '', s OTHERWISE;
> d = JOIN r BY x LEFT, c BY y;
> u2 = UNION ONSCHEMA d, s;
> e = FILTER u2 BY x == '';
> f = FILTER u2 BY x == 'm';
> u3 = UNION ONSCHEMA e, f;
> DUMP u3;
> {code}
> Which results in:
> {code}
> java.lang.IllegalArgumentException: VertexGroup must have at least 2 members
> at org.apache.tez.dag.api.VertexGroup.<init>(VertexGroup.java:77)
> at org.apache.tez.dag.api.DAG.createVertexGroup(DAG.java:202)
> at
> org.apache.pig.backend.hadoop.executionengine.tez.TezDagBuilder.visitTezOp(TezDagBuilder.java:396)
> at
> org.apache.pig.backend.hadoop.executionengine.tez.plan.TezOperator.visit(TezOperator.java:255)
> ...
> {code}
> This seems to be happening because {{UnionOptimizer}} is replacing a union
> with a vertex group and then optimizing away a predecessor union thus
> removing a node and resulting in a vertex group with one member.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
