Travis Woodruff created PIG-5082:
------------------------------------
Summary: Tez UnionOptimizer creates vertex group with one member
Key: PIG-5082
URL: https://issues.apache.org/jira/browse/PIG-5082
Project: Pig
Issue Type: Bug
Components: tez
Affects Versions: 0.16.0
Reporter: Travis Woodruff
Priority: Minor
This script results in a vertex group with one member:
{code}
a = LOAD '/tmp/empty.txt' USING PigStorage('\t') AS (x:chararray);
b = LOAD '/tmp/empty.txt' USING PigStorage('\t') AS (x:chararray);
c = LOAD '/tmp/empty.txt' USING PigStorage('\t') AS (y:chararray);
u1 = UNION ONSCHEMA a, b;
SPLIT u1 INTO r IF x != '', s OTHERWISE;
d = JOIN r BY x LEFT, c BY y;
u2 = UNION ONSCHEMA d, s;
e = FILTER u2 BY x == '';
f = FILTER u2 BY x == 'm';
u3 = UNION ONSCHEMA e, f;
DUMP u3;
{code}
Which results in:
{code}
java.lang.IllegalArgumentException: VertexGroup must have at least 2 members
at org.apache.tez.dag.api.VertexGroup.<init>(VertexGroup.java:77)
at org.apache.tez.dag.api.DAG.createVertexGroup(DAG.java:202)
at
org.apache.pig.backend.hadoop.executionengine.tez.TezDagBuilder.visitTezOp(TezDagBuilder.java:396)
at
org.apache.pig.backend.hadoop.executionengine.tez.plan.TezOperator.visit(TezOperator.java:255)
...
{code}
This seems to be happening because {{UnionOptimizer}} is replacing a union with
a vertex group and then optimizing away a predecessor union thus removing a
node and resulting in a vertex group with one member.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
