[
https://issues.apache.org/jira/browse/YARN-4678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15211403#comment-15211403
]
Brahma Reddy Battula commented on YARN-4678:
--------------------------------------------
bq.1) Understand why reserved resource + allocated resource could excess
queue's max capacity, maybe we can add a test to make sure it won't happen
Total= 9216 MB ( 9GB) where each NM is having 3GB ( total 3 NM's)
Configure a queueA with 10% and submit pi job with 2GB map memory and total-5
Containers request. Please check the following for same.
||Used||Usedcapacity||absoluteUsedCapacity||NM-Remaining||
|512 MB ( AM Conatiner Request)|0.55 ( 512 MB/ 921 MB)|0.055 ( 512MB /9216
MB)|{color:blue}NodeA{color}-2560MB|
|2560 MB ( Container allocation with
2GB)|2.7795875|0.2777778|{color:green}NodeB{color}-1024MB|
|4608 MB ( Container Reservation with
2GB)|5.0032573|0.5|{color:green}NodeB{color}-1024MB,{color:red}2048MB
Reserved{color}|
|6656 MB ( Container allocation with 2GB)|7.226927|0.7222222|NodeC-1024MB|
|8704 MB ( Container Reservation with 2GB)|
9.450597|0.9444444|NodeC-1024MB,{color:red}2048MB Reserved{color}||
|10752 MB ( Container allocation with 2GB)|
11.674267|1.1674267|{color:blue}NodeA{color}-512MB|
bq. 2) If we simply deduct reserved resources from used and show on the UI,
user could find cluster utilization is < 100 in most of the time, and it gonna
be hard to explain the reason of why it cannot reach 100%. The ideal solution
is that we can show reserved and allocated resources on the same bar with
different color.
I think, will it make confusion ..? as sunil mentioned we are giving metrics
for both resereved and used.. I feel, subracting from used should be ok.
> Cluster used capacity is > 100 when container reserved
> -------------------------------------------------------
>
> Key: YARN-4678
> URL: https://issues.apache.org/jira/browse/YARN-4678
> Project: Hadoop YARN
> Issue Type: Bug
> Reporter: Brahma Reddy Battula
> Assignee: Sunil G
> Attachments: 0001-YARN-4678.patch, 0002-YARN-4678.patch,
> 0003-YARN-4678.patch, reservedCapInClusterMetrics.png
>
>
> *Scenario:*
> * Start cluster with Three NM's each having 8GB (cluster memory:24GB).
> * Configure queues with elasticity and userlimitfactor=10.
> * disable pre-emption.
> * run two job with different priority in different queue at the same time
> ** yarn jar hadoop-mapreduce-examples-2.7.2.jar pi -Dyarn.app.priority=LOW
> -Dmapreduce.job.queuename=QueueA -Dmapreduce.map.memory.mb=4096
> -Dyarn.app.mapreduce.am.resource.mb=1536
> -Dmapreduce.job.reduce.slowstart.completedmaps=1.0 10 1000000000000
> ** yarn jar hadoop-mapreduce-examples-2.7.2.jar pi -Dyarn.app.priority=HIGH
> -Dmapreduce.job.queuename=QueueB -Dmapreduce.map.memory.mb=4096
> -Dyarn.app.mapreduce.am.resource.mb=1536 3 1000000000000
> * observe the cluster capacity which was used in RM web UI
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
