[
https://issues.apache.org/jira/browse/YARN-2041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14011844#comment-14011844
]
Nishkam Ravi commented on YARN-2041:
------------------------------------
Unlike FIFO, whose performance deteriorates consistently across multiple
benchmarks as value of yarn.nodemanager.resource.memory-mb is increased from
16GB to 40GB, Capacity scheduler performs well for all benchmarks except for
TeraValidate.
For TeraValidate in single-job mode:
Exec. time with Fair: 38 sec (yarn.nodemanager.resource.memory-mb = 16GB)
Exec. time with Fair: 38 sec (yarn.nodemanager.resource.memory-mb = 40GB)
Exec. time with Capacity: 51 sec (yarn.nodemanager.resource.memory-mb = 16GB)
Exec. time with Capacity: 100 sec (yarn.nodemanager.resource.memory-mb = 40GB)
Also, in multi-job mode, Capacity seems to be behaving like FIFO. Scheduling
one job at a time for execution.
> Hard to co-locate MR2 and Spark jobs on the same cluster in YARN
> ----------------------------------------------------------------
>
> Key: YARN-2041
> URL: https://issues.apache.org/jira/browse/YARN-2041
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: nodemanager
> Affects Versions: 2.3.0
> Reporter: Nishkam Ravi
>
> Performance of MR2 jobs falls drastically as YARN config parameter
> yarn.nodemanager.resource.memory-mb is increased beyond a certain value.
> Performance of Spark falls drastically as the value of
> yarn.nodemanager.resource.memory-mb is decreased beyond a certain value for a
> large data set.
> This makes it hard to co-locate MR2 and Spark jobs in YARN.
> The experiments are being conducted on a 6-node cluster. The following
> workloads are being run: TeraGen, TeraSort, TeraValidate, WordCount,
> ShuffleText and PageRank.
> Will add more details to this JIRA over time.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
