[
https://issues.apache.org/jira/browse/YARN-11108?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Juanjuan Tian updated YARN-11108:
----------------------------------
Description:
Found unexpected preemptions happen when hierarchy queues case, the issue is
that a sub queue can accept resource more than used+pending, leading to other
queues {color:#172b4d}IDEAL_ASSIGNED is smaller than used + pending, and
preemptions happen unexpectedly {color}
2022-04-02T01:11:12,973 DEBUG [SchedulingMonitor
(ProportionalCapacityPreemptionPolicy)]
org.apache.hadoop.yarn.server.resourcemanager.monitor.capacity.PreemptableResourceCalculator:
NAME: MSANRPAB PARTITION: persistent{color:#de350b} CUR: <memory:37934080,
vCores:8285{color}> PEN: <memory:0, vCores:0> TOTAL_PEN: <memory:0, vCores:0>
RESERVED: <memory:0, vCores:0> GAR: <memory:30720000, vCores:9571> NORM:
0.3424696922302246{color:#de350b} IDEAL_ASSIGNED: <memory:37934080,
vCores:8903>{color} IDEAL_PREEMPT: <memory:0, vCores:0, ports:null>
ACTUAL_PREEMPT: <memory:0, vCores:0, ports:null> UNTOUCHABLE: <memory:0,
vCores:0, ports:null> PREEMPTABLE: <memory:7214080, vCores:-1286, ports:null,
[<groupIndex:1, availableCpuCount:-36467,
from logs, we can see, {color:#de350b}IDEAL_ASSIGNED is bigger than CUR{color}
was:
Found unexpected preemptions happen when hierarchy queues case, the issue is
that a sub queue can accept resource more than used+pending, leading to other
queues {color:#172b4d}IDEAL_ASSIGNED is smaller than used + pending, and
preemptions happen unexpectedly {color}
2022-04-02T01:11:12,973 DEBUG [SchedulingMonitor
(ProportionalCapacityPreemptionPolicy)]
org.apache.hadoop.yarn.server.resourcemanager.monitor.capacity.PreemptableResourceCalculator:
NAME: MSANRPAB PARTITION: persistent{color:#de350b} CUR: <memory:37934080,
vCores:8285, ports:null{color}, [<groupIndex:1, availableCpuCount:-36467,
reservedAffinity:\\{6, 8, 9, 10, 11, 15, 19, 20, 22, 24, 28}>]> PEN: <memory:0,
vCores:0, ports:null> TOTAL_PEN: <memory:0, vCores:0, ports:null> RESERVED:
<memory:0, vCores:0, ports:null> GAR: <memory:30720000, vCores:9571,
ports:null> NORM: 0.3424696922302246{color:#de350b} IDEAL_ASSIGNED:
<memory:37934080, vCores:8903, ports:null>{color} IDEAL_PREEMPT: <memory:0,
vCores:0, ports:null> ACTUAL_PREEMPT: <memory:0, vCores:0, ports:null>
UNTOUCHABLE: <memory:0, vCores:0, ports:null> PREEMPTABLE: <memory:7214080,
vCores:-1286, ports:null, [<groupIndex:1, availableCpuCount:-36467,
reservedAffinity:\\{6, 8, 9, 10, 11, 15, 19, 20, 22, 24, 28}>]> BONUS_WEIGHT:
-1.0
> Unexpected preemptions happen when hierarchy queues case
> --------------------------------------------------------
>
> Key: YARN-11108
> URL: https://issues.apache.org/jira/browse/YARN-11108
> Project: Hadoop YARN
> Issue Type: Improvement
> Affects Versions: 2.9.2
> Reporter: Juanjuan Tian
> Assignee: Juanjuan Tian
> Priority: Major
> Attachments: image-2022-04-06-16-29-57-871.png
>
>
> Found unexpected preemptions happen when hierarchy queues case, the issue is
> that a sub queue can accept resource more than used+pending, leading to other
> queues {color:#172b4d}IDEAL_ASSIGNED is smaller than used + pending, and
> preemptions happen unexpectedly {color}
>
> 2022-04-02T01:11:12,973 DEBUG [SchedulingMonitor
> (ProportionalCapacityPreemptionPolicy)]
> org.apache.hadoop.yarn.server.resourcemanager.monitor.capacity.PreemptableResourceCalculator:
> NAME: MSANRPAB PARTITION: persistent{color:#de350b} CUR: <memory:37934080,
> vCores:8285{color}> PEN: <memory:0, vCores:0> TOTAL_PEN: <memory:0, vCores:0>
> RESERVED: <memory:0, vCores:0> GAR: <memory:30720000, vCores:9571> NORM:
> 0.3424696922302246{color:#de350b} IDEAL_ASSIGNED: <memory:37934080,
> vCores:8903>{color} IDEAL_PREEMPT: <memory:0, vCores:0, ports:null>
> ACTUAL_PREEMPT: <memory:0, vCores:0, ports:null> UNTOUCHABLE: <memory:0,
> vCores:0, ports:null> PREEMPTABLE: <memory:7214080, vCores:-1286, ports:null,
> [<groupIndex:1, availableCpuCount:-36467,
>
> from logs, we can see, {color:#de350b}IDEAL_ASSIGNED is bigger than
> CUR{color}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
