Hi Omkar,
>(which has 40 containers slots.) >> for total cluster?
Yes, it was just an hypotetical value though.
Below are my real configurations.
>1) yarn-site.xml -> what is the resource memory configured for per node?
12288mb
>2) yarn-site.xml -> what is the minimum resource allocation for the
cluster?
1024mb min
12288mb max
I also have those memory configurations in mapred-site.xml :
<property>
<name>mapreduce.map.memory.mb</name>
<value>5000</value>
</property>
<property>
<name>mapreduce.map.java.opts</name>
<value>-Xmx4g -Djava.awt.headless=true</value>
</property>
<property>
<name>mapreduce.reduce.memory.mb</name>
<value>5000</value>
</property>
<property>
<name>mapreduce.reduce.java.opts</name>
<value>-Xmx4g -Djava.awt.headless=true</value>
</property>
>3) yarn-resource-manager-log (while starting resource manager "export
YARN_ROOT_LOGGER=DEBUG,RFA").. I am looking for debug logs..
The resulting log is really verbose. Are you searching for something in
particular?
>4) On RM UI how much total cluster memory is reported (how many total
nodes). ( RM UI click on Cluster)
So I have 58 active nodes and total memory reported is 696GB which is
58x12 as expected.
I have 93 containers running instead of 116 I would expect (my job has
2046 maps so it could use all 116 containers).
Here is a copy past of what I have in the scheduler tab:
*Queue State: *
RUNNING
*Used Capacity: *
99.4%
*Absolute Capacity: *
100.0%
*Absolute Max Capacity: *
100.0%
*Used Resources: *
*Num Active Applications: *
1
*Num Pending Applications: *
0
*Num Containers: *
139
*Max Applications: *
10000
*Max Applications Per User: *
10000
*Max Active Applications: *
70
*Max Active Applications Per User: *
70
*Configured Capacity: *
100.0%
*Configured Max Capacity: *
100.0%
*Configured Minimum User Limit Percent: *
100%
*Configured User Limit Factor: *
1.0
*Active users: *
xxx <Memory: 708608 (100.00%), vCores: 139 (100.00%), Active Apps: 1,
Pending Apps: 0>
I don't know where the 139 containers value is comming from.
>5) which scheduler you are using? Capacity/Fair/FIFO
I did not set yarn.resourcemanager.scheduler.class so apparently the
default is Capacity.
>6) have you configured any user limits/ queue capacity? (please add
details).
No.
>7) All requests you are making at same priority or with different priorities?
(Ideally it will not matter but want to know).
I don't set any priority.
Thanks for your help.
Antoine Vandecreme
On Friday, September 20, 2013 12:20:38 PM Omkar Joshi wrote:
> Hi,
>
> few more questions
>
> (which has 40 containers slots.) >> for total cluster? Please give below
> details
>
> for cluster
> 1) yarn-site.xml -> what is the resource memory configured for per node?
> 2) yarn-site.xml -> what is the minimum resource allocation for the
cluster?
> 3) yarn-resource-manager-log (while starting resource manager "export
> YARN_ROOT_LOGGER=DEBUG,RFA").. I am looking for debug logs..
> 4) On RM UI how much total cluster memory is reported (how many total
> nodes). ( RM UI click on Cluster)
> 5) which scheduler you are using? Capacity/Fair/FIFO
> 6) have you configured any user limits/ queue capacity? (please add
> details).
> 7) All requests you are making at same priority or with different
> priorities? (Ideally it will not matter but want to know).
>
> Please let us know all the above details. Thanks.
>
>
> Thanks,
> Omkar Joshi
> *Hortonworks Inc.* <http://www.hortonworks.com>
>
>
> On Fri, Sep 20, 2013 at 6:55 AM, Antoine Vandecreme <
>
> [email protected]> wrote:
> > Hello Omkar,
> >
> > Thanks for your reply.
> >
> > Yes, all 4 points are corrects.
