On each node you can configure how much memory is available for containers to run. On the other hand, for each application you can configure how large containers should be. For MR apps, you can separately set mappers, reducers, and the app master itself.
Yarn will detemine through scheduling rules and depending on locality where tasks are run. One app has one container size (per respective category map, reduce, AM) that is not driven by nodes. Available node memory divided by task size will determine how many tasks run on each node. There are minimum and maximum container sizes, so you can avoid running crazy things such as 1K 1MB containers for example. Hope that helps, Joep On Thu, Jun 4, 2015 at 6:48 AM, paco <[email protected]> wrote: > > Hello, > > Recently I have increased my physical cluster. I have two kind of nodes: > > Type 1: > RAM: 24 GB > 12 cores > > Type 2: > RAM: 64 GB > 12 cores > > Theses nodes are in the same physical rack. I would like to configure it > to use 12 container per node, in nodes of type 1 each mapper has 1.8GB > (22GB / 12 cores = 1.8GB), in nodes of kind 2 each mapper will has 5.3GB > (60/12). Is it possible? > > I have configured so: > > nodes type 1(slaves): > <property> > <name>yarn.nodemanager.resource.memory-mb</name> > <value>22000</value> > </property> > > <property> > <name>mapreduce.map.memory.mb</name> > <value>1800</value> > </property> > <property> > <name>mapred.map.child.java.opts</name> > <value>-Xmx1800m</value> > </property> > > > > nodes type 2(slaves): > <property> > <name>yarn.nodemanager.resource.memory-mb</name> > <value>60000</value> > </property> > > <property> > <name>mapreduce.map.memory.mb</name> > <value>5260</value> > </property> > <property> > <name>mapred.map.child.java.opts</name> > <value>-Xmx5260m</value> > </property> > > > > Hadoop is creating mapper with 1 GB of memory like: > > Nodes of kind 1: > 20GB/1GB = 20 container which it is executing with -Xmx1800 > > Nodes of kind 2: > 60GB/1GB = 60 container which it is executing with -Xmx5260 > > > Thanks! > > >
