[
https://issues.apache.org/jira/browse/PIG-4846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15251565#comment-15251565
]
liyunzhang_intel commented on PIG-4846:
---------------------------------------
[~rohini]: Thanks for your comment. now i use new configuration as following
to test and check whether generate the output successfully or not after test.
{code}
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
<property>
<name>mapred.child.java.opts</name>
<value>-Xmx1792m</value>
</property>
<property>
<name>mapreduce.map.memory.mb</name>
<value>2048</value>
</property>
<property>
<name>mapreduce.reduce.memory.mb</name>
<value>2048</value>
</property>
<property>
<name>mapreduce.task.io.sort.mb</name>
<value>1024</value>
</property>
</configuration>
{code}
The result of mr has a lot improvement this time, run faster than what in
spark. So i investigate a little and use a patch from PIG-4797 to retest. Now
the result between mr and spark is:
||Script||mr||spark||
|L_1|51|53|
|L_2|37|45|
|L_3|92|49|
|L_4|51|48|
|L_5|51|47|
|L_6|51|50|
|L_7|46|46|
|L_8|26|46|
|L_9|89|50|
|L_10|89|51|
|L_11|326|64|
|L_12|46|54|
|L_13|40|46|
|L_14|56|47|
|L_15|50|47|
|L_16|50|50|
|L_17|82|54|
> Use pigmix to test the performance of pig on spark
> --------------------------------------------------
>
> Key: PIG-4846
> URL: https://issues.apache.org/jira/browse/PIG-4846
> Project: Pig
> Issue Type: Sub-task
> Components: spark
> Reporter: liyunzhang_intel
> Assignee: liyunzhang_intel
> Fix For: spark-branch
>
> Attachments: PIG-4846.patch, PIG-4846_1.patch
>
>
> We can compare the performance between mr and spark mode by pigmix.
> The introduction of pigmix is
> https://cwiki.apache.org/confluence/display/PIG/PigMix.
> PIG-4846.patch is to make pigmix run by specied exectype.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
