[
https://issues.apache.org/jira/browse/PIG-3835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13957137#comment-13957137
]
Daniel Dai commented on PIG-3835:
---------------------------------
Thanks for a quick respons. PIG-3835-addendum-1.patch works for me. The script
throw exception pass now, though the correct result will depend on TEZ-1003.
rmf output
a = load '1.txt' as (x:int, y:chararray);
b = load '2.txt' as (y:chararray, x:int);
c = union onschema a, b;
d = group c by x;
-- store c into 'output';
store d into 'output';
+1
> Improve performance of union
> ----------------------------
>
> Key: PIG-3835
> URL: https://issues.apache.org/jira/browse/PIG-3835
> Project: Pig
> Issue Type: Sub-task
> Components: tez
> Affects Versions: tez-branch
> Reporter: Cheolsoo Park
> Assignee: Rohini Palaniswamy
> Fix For: tez-branch
>
> Attachments: PIG-3835-2.patch, PIG-3835-3.patch,
> PIG-3835-Initial-1.patch, PIG-3835-addendum-1.patch
>
>
> PIG-3743 implements union using VertexGroup. But there are a couple of
> optimizations that we can apply to it.
> * Union followed by store
> Union is a blocking operator meaning that a new vertex is added for its
> succeeding operators. But if there is only one store in the succeeding
> vertex, MROutput could be directly attached to VertexGroup instead of adding
> a new vertex for it. Then, each union source vertex will write directly to
> the destination, and therefore, it will be faster.
> * Replace POLocalRearrangeTez with POValueOutputTez
> Union uses POLocalRearrange by setting the whole record as key. But since
> union only needs to partition records evenly across tasks, it might make more
> sense to use POValueOutputTez with RR partitioner instead.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
