Nishkam Ravi created YARN-2041:
----------------------------------
Summary: Hard to co-locate MR2 and Spark jobs on the same cluster
in YARN
Key: YARN-2041
URL: https://issues.apache.org/jira/browse/YARN-2041
Project: Hadoop YARN
Issue Type: Improvement
Components: nodemanager
Affects Versions: 2.3.0
Reporter: Nishkam Ravi
Fix For: 2.4.0, 2.3.0
Performance of MR2 jobs falls drastically as YARN config parameter
yarn.nodemanager.resource.memory-mb is increased beyond a certain value.
Performance of Spark falls drastically as the value of
yarn.nodemanager.resource.memory-mb is decreased beyond a certain value for a
large data set.
This makes it hard to co-locate MR2 and Spark jobs in YARN.
The experiments are being conducted on a 6-node cluster. The following
workloads are being run: TeraGen, TeraSort, TeraValidate, WordCount,
ShuffleText and PageRank.
Will add more details to this JIRA over time.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
