[
https://issues.apache.org/jira/browse/PIG-4846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15239246#comment-15239246
]
Xuefu Zhang commented on PIG-4846:
----------------------------------
Your machine seems having abundant cores but scarce memory. I suggest the
following:
YARN configuration:
{code}
yarn.nodemanager.resource.memory-mb=56G
yarn.nodemanger.resource.cpu-vcores=28
{code}
Spark configurations:
{code}
spark.executor.cores=4
spark.executor.memory=6.4G
spark.yarn.executor.memoryOverhead=1.6G
spark.driver.memory=2G
spark.yarn.driver.memoryOverhead=400M
spark.executor.instances=7
{code}
Please note that the numbers might need to be converted to the unit of
individual property. For instance, .memory takes bytes while memoryOverhead
takes MB.
> Use pigmix to test the performance of pig on spark
> --------------------------------------------------
>
> Key: PIG-4846
> URL: https://issues.apache.org/jira/browse/PIG-4846
> Project: Pig
> Issue Type: Sub-task
> Components: spark
> Reporter: liyunzhang_intel
> Assignee: liyunzhang_intel
> Fix For: spark-branch
>
> Attachments: PIG-4846.patch, PIG-4846_1.patch
>
>
> We can compare the performance between mr and spark mode by pigmix.
> The introduction of pigmix is
> https://cwiki.apache.org/confluence/display/PIG/PigMix.
> PIG-4846.patch is to make pigmix run by specied exectype.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
