Hi
If you are using CapacityScheduler, can you try using
DominantResourceCalculator i.e configuring below property value in
capacity-scheduler.xml file.
<property>
<name>yarn.scheduler.capacity.resource-calculator</name>
<value>org.apache.hadoop.yarn.util.resource. DominantResourceCalculator
</value>
</property>
The basic Idea it works is as follows ‘if user A runs CPU-heavy tasks and user
B runs memory-heavy tasks, it attempts to equalize CPU share of user A with
Memory-share of user B’
See Java Doc
https://apache.googlesource.com/hadoop-common/+/60e3b885ba8344d9f448202f5f2c290b5606ff8f/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/resource/DominantResourceCalculator.java
I think this may help you!!!
Thanks & Regards
Rohith Sharma K S
From: twinkle sachdeva [mailto:[email protected]]
Sent: 26 February 2015 14:05
To: USers Hadoop
Subject: Node manager contributing to one queue's resources
Hi,
I have to run two kind of applications, one requiring less cores but more
memory ( Application_High_Mem) and another application which requires more
cores but less memory ( Application_High_Core).
I can use specific queues to submit them to, but that can lead to one node
contributing to only one one such application and having some part of resources
as idle.
Is there a way, let's say extending concept of queues at node manager level to
do this or some other way, in which i can achieve it in YARN?
Thanks,
Twinkle
