[
https://issues.apache.org/jira/browse/PIG-4846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15247849#comment-15247849
]
Rohini Palaniswamy commented on PIG-4846:
-----------------------------------------
You have allotted 1G heap and 1G io.sort.mb. I am surprised how the job even
ran instead of hitting OOM as there is no space for anything else. Please set
mapred.child.java.opts to -Xmx1792m. 1G difference between container size and
heap size is just wasting resource without using them. Also please check that
the number of records match and bytes output closely match from job stats
printed (Successfully stored ...). I have the habit of doing that everytime I
run Pigmix as we don't do data comparison for Pigmix like e2e tests.
L9 and L10 are order by. Good to see that it is faster.
> Use pigmix to test the performance of pig on spark
> --------------------------------------------------
>
> Key: PIG-4846
> URL: https://issues.apache.org/jira/browse/PIG-4846
> Project: Pig
> Issue Type: Sub-task
> Components: spark
> Reporter: liyunzhang_intel
> Assignee: liyunzhang_intel
> Fix For: spark-branch
>
> Attachments: PIG-4846.patch, PIG-4846_1.patch
>
>
> We can compare the performance between mr and spark mode by pigmix.
> The introduction of pigmix is
> https://cwiki.apache.org/confluence/display/PIG/PigMix.
> PIG-4846.patch is to make pigmix run by specied exectype.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
