Wangda Tan commented on YARN-2305:

Hi [~sunilg],
Thanks for taking this issue,
I think there're two issues in your screenshot,
1) Root queue usage above 100%
It is possible that queue used resource is larger than its guaranteed resource 
because of container reservation. We may need show reserved resource and used 
resource separately in our web UI. I encountered a similar problem in YARN-2285 

2) Total cluster memory showing on web UI is different from 
This seems a new issue to me, memory showing on web UI is 
usedMemory+availableMemory of root queue. I feel like 
CSQueueUtils.updateQueueStatistics has some issues when we reserve container in 
LeafQueue. Hope to get more thoughts in your side.


> When a container is in reserved state then total cluster memory is displayed 
> wrongly.
> -------------------------------------------------------------------------------------
>                 Key: YARN-2305
>                 URL: https://issues.apache.org/jira/browse/YARN-2305
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: resourcemanager
>    Affects Versions: 2.4.1
>            Reporter: J.Andreina
>            Assignee: Sunil G
>         Attachments: Capture.jpg
> ENV Details:
> =========  
>          3 queues  :  a(50%),b(25%),c(25%) ---> All max utilization is set to 
> 100
>          2 Node cluster with total memory as 16GB
> TestSteps:
> =========
>           Execute following 3 jobs with different memory configurations for 
> Map , reducer and AM task
>                   ./yarn jar wordcount-sleep.jar -Dmapreduce.job.queuename=a 
> -Dwordcount.map.sleep.time=2000 -Dmapreduce.map.memory.mb=2048 
> -Dyarn.app.mapreduce.am.resource.mb=1024 -Dmapreduce.reduce.memory.mb=2048 
> /dir8 /preempt_85 (application_1405414066690_0023)
>                  ./yarn jar wordcount-sleep.jar -Dmapreduce.job.queuename=b 
> -Dwordcount.map.sleep.time=2000 -Dmapreduce.map.memory.mb=2048 
> -Dyarn.app.mapreduce.am.resource.mb=2048 -Dmapreduce.reduce.memory.mb=2048 
> /dir2 /preempt_86 (application_1405414066690_0025)
>                  ./yarn jar wordcount-sleep.jar -Dmapreduce.job.queuename=c 
> -Dwordcount.map.sleep.time=2000 -Dmapreduce.map.memory.mb=1024 
> -Dyarn.app.mapreduce.am.resource.mb=1024 -Dmapreduce.reduce.memory.mb=1024 
> /dir2 /preempt_62
> Issue
> =====
>               when 2GB memory is in reserved state  totoal memory is shown as 
> 15GB and used as 15GB  ( while total memory is 16GB)

This message was sent by Atlassian JIRA

Reply via email to