[
https://issues.apache.org/jira/browse/PIG-3835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13957838#comment-13957838
]
Cheolsoo Park commented on PIG-3835:
------------------------------------
FYI-
# This patch makes my test job run about 7% faster.
|| ||w/o PIG-3835||w/ PIG-3835||
|HH:mm:ss|00:15:32|00:14:33|
# Regarding EmptyWritableComparator, 0 is faster than -1 and 1.
|| ||-1||0||1||
|HH:mm:ss|00:16:21|00:14:30|0:15:10|
> Improve performance of union
> ----------------------------
>
> Key: PIG-3835
> URL: https://issues.apache.org/jira/browse/PIG-3835
> Project: Pig
> Issue Type: Sub-task
> Components: tez
> Affects Versions: tez-branch
> Reporter: Cheolsoo Park
> Assignee: Rohini Palaniswamy
> Fix For: tez-branch
>
> Attachments: PIG-3835-2.patch, PIG-3835-3.patch,
> PIG-3835-Initial-1.patch, PIG-3835-addendum-1.patch
>
>
> PIG-3743 implements union using VertexGroup. But there are a couple of
> optimizations that we can apply to it.
> * Union followed by store
> Union is a blocking operator meaning that a new vertex is added for its
> succeeding operators. But if there is only one store in the succeeding
> vertex, MROutput could be directly attached to VertexGroup instead of adding
> a new vertex for it. Then, each union source vertex will write directly to
> the destination, and therefore, it will be faster.
> * Replace POLocalRearrangeTez with POValueOutputTez
> Union uses POLocalRearrange by setting the whole record as key. But since
> union only needs to partition records evenly across tasks, it might make more
> sense to use POValueOutputTez with RR partitioner instead.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
