[
https://issues.apache.org/jira/browse/PIG-4846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15237598#comment-15237598
]
Xuefu Zhang commented on PIG-4846:
----------------------------------
How many cores do you have on the machine? What's the value for
yarn.nodemanager.resource.cpu-vcores?
Besides spark.executor.memory, you probably want to set values for the
following:
spark.executor.cores
spark.yarn.executor.memoryOverhead
spark.driver.memory
spark.yarn.driver.memoryOverhead
spark.executor.instances
Once I know the value of yarn.nodemanager.resource.cpu-vcores, I can come up
with a set of configuration values for you to test.
> Use pigmix to test the performance of pig on spark
> --------------------------------------------------
>
> Key: PIG-4846
> URL: https://issues.apache.org/jira/browse/PIG-4846
> Project: Pig
> Issue Type: Sub-task
> Components: spark
> Reporter: liyunzhang_intel
> Assignee: liyunzhang_intel
> Fix For: spark-branch
>
> Attachments: PIG-4846.patch, PIG-4846_1.patch
>
>
> We can compare the performance between mr and spark mode by pigmix.
> The introduction of pigmix is
> https://cwiki.apache.org/confluence/display/PIG/PigMix.
> PIG-4846.patch is to make pigmix run by specied exectype.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
