[
https://issues.apache.org/jira/browse/PIG-4846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15247041#comment-15247041
]
liyunzhang_intel commented on PIG-4846:
---------------------------------------
[~rohini]: have modified the mapred-site.xml as your suggestion
{code}
<configuration>
....
<property>
<name>mapred.child.java.opts</name>
<value>-Xmx1024m</value>
</property>
<property>
<name>mapreduce.map.memory.mb</name>
<value>2048</value>
</property>
<property>
<name>mapreduce.reduce.memory.mb</name>
<value>2048</value>
</property>
<property>
<name>mapreduce.task.io.sort.mb</name>
<value>1024</value>
</property>
</configuration>
{code}
Following are the new result of mr and spark(yarn-client) mode:
||Script||mr||spark||
|L_1|67|55|
|L_2|340|45|
|L_3|66|337|
|L_4|60|48|
|L_5|65|338|
|L_6|60|48|
|L_7|61|210|
|L_8|60|46|
|L_9|233|51|
|L_10|233|51|
|L_11|126|65|
|L_12|60|54|
|L_13|65|212|
|L_14|75|45|
|L_15|66|210|
|L_16|65|207|
|L_17|80|54|
Most scripts run faster in spark mode except L3,L5,L7,L13,L15,L16
> Use pigmix to test the performance of pig on spark
> --------------------------------------------------
>
> Key: PIG-4846
> URL: https://issues.apache.org/jira/browse/PIG-4846
> Project: Pig
> Issue Type: Sub-task
> Components: spark
> Reporter: liyunzhang_intel
> Assignee: liyunzhang_intel
> Fix For: spark-branch
>
> Attachments: PIG-4846.patch, PIG-4846_1.patch
>
>
> We can compare the performance between mr and spark mode by pigmix.
> The introduction of pigmix is
> https://cwiki.apache.org/confluence/display/PIG/PigMix.
> PIG-4846.patch is to make pigmix run by specied exectype.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
