Nishkam Ravi commented on YARN-2041:

Sure. There seem to be multiple issues. Two seem to clearly stand out:

1. Performance degrades with FIFO for large values of memory-mb in both 
single-job and multi-job mode. Observed for multiple benchmarks including 
TeraSort, TeraValidate, TeraGen, WordCount, ShuffleText. Issue: FIFO seems to 
be allocating too many jobs at once on a single node.
2. Performance with Capacity scheduler suffers for large values of memory-mb 
only for TeraValidate (in single-job mode). Issue: why does Capacity scheduler 
regress for TeraValidate? 

> Hard to co-locate MR2 and Spark jobs on the same cluster in YARN
> ----------------------------------------------------------------
>                 Key: YARN-2041
>                 URL: https://issues.apache.org/jira/browse/YARN-2041
>             Project: Hadoop YARN
>          Issue Type: Improvement
>          Components: nodemanager
>    Affects Versions: 2.3.0
>            Reporter: Nishkam Ravi
> Performance of MR2 jobs falls drastically as YARN config parameter 
> yarn.nodemanager.resource.memory-mb  is increased beyond a certain value. 
> Performance of Spark falls drastically as the value of 
> yarn.nodemanager.resource.memory-mb is decreased beyond a certain value for a 
> large data set.
> This makes it hard to co-locate MR2 and Spark jobs in YARN.
> The experiments are being conducted on a 6-node cluster. The following 
> workloads are being run: TeraGen, TeraSort, TeraValidate, WordCount, 
> ShuffleText and PageRank.
> Will add more details to this JIRA over time.

This message was sent by Atlassian JIRA

Reply via email to