[jira] [Commented] (YARN-10624) Support max queues limit configuration in new auto created queue, consistent with old auto created.
[
https://issues.apache.org/jira/browse/YARN-10624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17284097#comment-17284097
]
Hadoop QA commented on YARN-10624:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
12s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} No case conflicting files
found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch does not contain any
@author tags. {color} |
| {color:green}+1{color} | {color:green} {color} | {color:green} 0m 0s{color}
| {color:green}test4tests{color} | {color:green} The patch appears to include 1
new or modified test files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 22m
41s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
0s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
50s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
45s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
52s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
16m 39s{color} | {color:green}{color} | {color:green} branch has no errors when
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
41s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
36s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 1m
50s{color} | {color:blue}{color} | {color:blue} Used deprecated FindBugs
config; considering switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
48s{color} | {color:green}{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
48s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
54s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
54s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
44s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
44s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 40s{color} |
{color:orange}https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/613/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt{color}
| {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 1 new + 94 unchanged - 0 fixed = 95 total (was 94) {color}
|
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
47s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch has no whitespace
issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
15m 5s{color} | {color:green}{color} | {color:green} patch has no errors when
building and testing our client artifacts. {color} |
|
[jira] [Updated] (YARN-10624) Support max queues limit configuration in new auto created queue, consistent with old auto created.
[ https://issues.apache.org/jira/browse/YARN-10624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qi Zhu updated YARN-10624: -- Summary: Support max queues limit configuration in new auto created queue, consistent with old auto created. (was: Support max leaf queues limit configuration, consistent with old auto created.) > Support max queues limit configuration in new auto created queue, consistent > with old auto created. > --- > > Key: YARN-10624 > URL: https://issues.apache.org/jira/browse/YARN-10624 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Qi Zhu >Assignee: Qi Zhu >Priority: Major > > Since old created leaf queue has the max leaf queues limit, i think we also > should support this in new auto created queue. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10624) Support max queues limit configuration in new auto created queue, consistent with old auto created.
[ https://issues.apache.org/jira/browse/YARN-10624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qi Zhu updated YARN-10624: -- Description: Since old created leaf queue has the max leaf queues limit, i think we also should support this in new auto created queue, both the auto created leaf and the auto created parent need limits. (was: Since old created leaf queue has the max leaf queues limit, i think we also should support this in new auto created queue.) > Support max queues limit configuration in new auto created queue, consistent > with old auto created. > --- > > Key: YARN-10624 > URL: https://issues.apache.org/jira/browse/YARN-10624 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Qi Zhu >Assignee: Qi Zhu >Priority: Major > > Since old created leaf queue has the max leaf queues limit, i think we also > should support this in new auto created queue, both the auto created leaf and > the auto created parent need limits. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-10532) Capacity Scheduler Auto Queue Creation: Allow auto delete queue when queue is not being used
[ https://issues.apache.org/jira/browse/YARN-10532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17283691#comment-17283691 ] Qi Zhu edited comment on YARN-10532 at 2/13/21, 2:11 AM: - !image-2021-02-12-21-32-02-267.png|width=1085,height=764! cc [~gandras] [~snemeth] [~ztang] [~bteke] [~shuzirra] [~pbacsko] [~epayne] I have also confirmed it in my test cluster, i think this is very import for Auto created queue, if you could help review the latest patch? Thanks. was (Author: zhuqi): !image-2021-02-12-21-32-02-267.png|width=1085,height=764! cc [~gandras] [~snemeth] [~ztang] [~epayne] I have also confirmed it in my test cluster, i think this is very import for Auto created queue, if you could help review the latest patch? Thanks. > Capacity Scheduler Auto Queue Creation: Allow auto delete queue when queue is > not being used > > > Key: YARN-10532 > URL: https://issues.apache.org/jira/browse/YARN-10532 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan >Assignee: Qi Zhu >Priority: Major > Attachments: YARN-10532.001.patch, YARN-10532.002.patch, > YARN-10532.003.patch, YARN-10532.004.patch, YARN-10532.005.patch, > YARN-10532.006.patch, YARN-10532.007.patch, YARN-10532.008.patch, > YARN-10532.009.patch, YARN-10532.010.patch, YARN-10532.011.patch, > YARN-10532.012.patch, YARN-10532.013.patch, YARN-10532.014.patch, > YARN-10532.015.patch, YARN-10532.016.patch, YARN-10532.017.patch, > YARN-10532.018.patch, YARN-10532.019.patch, YARN-10532.020.patch, > image-2021-02-12-21-32-02-267.png > > > It's better if we can delete auto-created queues when they are not in use for > a period of time (like 5 mins). It will be helpful when we have a large > number of auto-created queues (e.g. from 500 users), but only a small subset > of queues are actively used. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-10532) Capacity Scheduler Auto Queue Creation: Allow auto delete queue when queue is not being used
[ https://issues.apache.org/jira/browse/YARN-10532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17283691#comment-17283691 ] Qi Zhu edited comment on YARN-10532 at 2/12/21, 2:15 PM: - !image-2021-02-12-21-32-02-267.png|width=1085,height=764! cc [~gandras] [~snemeth] [~ztang] [~epayne] I have also confirmed it in my test cluster, i think this is very import for Auto created queue, if you could help review the latest patch? Thanks. was (Author: zhuqi): !image-2021-02-12-21-32-02-267.png|width=1085,height=764! cc [~gandras] [~snemeth] [~ztang] [~epayne] I have also confirmed it in my test cluster, i think this is very import for Auto created leaf queue, if you could help review the latest patch? Thanks. > Capacity Scheduler Auto Queue Creation: Allow auto delete queue when queue is > not being used > > > Key: YARN-10532 > URL: https://issues.apache.org/jira/browse/YARN-10532 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan >Assignee: Qi Zhu >Priority: Major > Attachments: YARN-10532.001.patch, YARN-10532.002.patch, > YARN-10532.003.patch, YARN-10532.004.patch, YARN-10532.005.patch, > YARN-10532.006.patch, YARN-10532.007.patch, YARN-10532.008.patch, > YARN-10532.009.patch, YARN-10532.010.patch, YARN-10532.011.patch, > YARN-10532.012.patch, YARN-10532.013.patch, YARN-10532.014.patch, > YARN-10532.015.patch, YARN-10532.016.patch, YARN-10532.017.patch, > YARN-10532.018.patch, YARN-10532.019.patch, YARN-10532.020.patch, > image-2021-02-12-21-32-02-267.png > > > It's better if we can delete auto-created queues when they are not in use for > a period of time (like 5 mins). It will be helpful when we have a large > number of auto-created queues (e.g. from 500 users), but only a small subset > of queues are actively used. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-10532) Capacity Scheduler Auto Queue Creation: Allow auto delete queue when queue is not being used
[ https://issues.apache.org/jira/browse/YARN-10532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17283691#comment-17283691 ] Qi Zhu edited comment on YARN-10532 at 2/12/21, 2:15 PM: - !image-2021-02-12-21-32-02-267.png|width=1085,height=764! cc [~gandras] [~snemeth] [~ztang] [~epayne] I have also confirmed it in my test cluster, i think this is very import for Auto created leaf queue, if you could help review the latest patch? Thanks. was (Author: zhuqi): !image-2021-02-12-21-32-02-267.png|width=1085,height=764! cc [~gandras] [~snemeth] [~ztang] [~epayne] I have also confirmed it in my test cluster, if you could help review the latest patch? Thanks. > Capacity Scheduler Auto Queue Creation: Allow auto delete queue when queue is > not being used > > > Key: YARN-10532 > URL: https://issues.apache.org/jira/browse/YARN-10532 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan >Assignee: Qi Zhu >Priority: Major > Attachments: YARN-10532.001.patch, YARN-10532.002.patch, > YARN-10532.003.patch, YARN-10532.004.patch, YARN-10532.005.patch, > YARN-10532.006.patch, YARN-10532.007.patch, YARN-10532.008.patch, > YARN-10532.009.patch, YARN-10532.010.patch, YARN-10532.011.patch, > YARN-10532.012.patch, YARN-10532.013.patch, YARN-10532.014.patch, > YARN-10532.015.patch, YARN-10532.016.patch, YARN-10532.017.patch, > YARN-10532.018.patch, YARN-10532.019.patch, YARN-10532.020.patch, > image-2021-02-12-21-32-02-267.png > > > It's better if we can delete auto-created queues when they are not in use for > a period of time (like 5 mins). It will be helpful when we have a large > number of auto-created queues (e.g. from 500 users), but only a small subset > of queues are actively used. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10621) GPU management using OpenCL instead of vendor-specific solutions
[
https://issues.apache.org/jira/browse/YARN-10621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17283717#comment-17283717
]
Adam Antal commented on YARN-10621:
---
Thanks for bringing this issue to the community!
>From the JIRA details I assume that you're using version 3.1.1, which has not
>yet included the {{{DevicePlugin}} interface. This interface has been added to
>the code for the very same purpose: discovering custom resources provided by
>these plugin - just like the Nvidia GPUs. For more information look at the
>umbrella jira: YARN-8851.
I don't know if the changes you've been working on are based on this work, but
it's the recommended way from 3.3.0 on. How is that aligns with you effort?
> GPU management using OpenCL instead of vendor-specific solutions
>
>
> Key: YARN-10621
> URL: https://issues.apache.org/jira/browse/YARN-10621
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: nodemanager, yarn
>Reporter: Sotiris Niarchos
>Priority: Minor
>
> As part of the [E2Data research project|https://e2data.eu/], we at the
> [Institute of Communication and Computer Systems
> (ICCS)|https://www.iccs.gr/en/?noredirect=en_US] of the National Technical
> University of Athens, Greece, have been working on a modified version of
> Hadoop Yarn where the GPU devices that are available in the underlying
> cluster are discovered via a Java wrapper of the OpenCL framework API (namely
> [JOCL|https://github.com/gpu/JOCL]), instead of vendor-specific binaries.
> In other words, we have shifted towards *a more uniform and high-level
> handling of GPUs as "OpenCL-enabled" devices*. This way, we manage to
> *decouple GPU discovery/management from vendor-specific technicalities*;
> every GPU, no matter the vendor, is the same for E2Data YARN (more
> specifically, for the {{NodeManager}} component), provided that the OpenCL
> runtime and drivers for the GPU(s) of interest are installed on the
> respective node(s) of the cluster.
> This way, we *managed to use GPUs other than NVIDIA* (which are the only ones
> officially supported via the {{nvidia-smi}} binary) with minimal additional
> effort, after our initial changes.
> Ultimately, our goal is to *unify every processing unit* that YARN can
> possible utilize (CPU cores, GPUs, FPGAs) *behind a common, simple,
> high-level interface; that of the OpenCL-enabled device*.
> The only drawback of our approach is that vendor-specific info regarding the
> GPUs is lost (e.g. temperature). We believe, however, that the lost
> information is not necessary for YARN; everything that Hadoop needs in order
> to discover and handle GPU devices is provided by OpenCL.
> This is just a proposition/a prompt for discussion for the time being. This
> modified version is a work in progress. We consider community feedback
> regarding the core concept (and the fact that it may constitute a paradigm
> shift for YARN) crucial before attaching any patch file and diving into more
> (technical) details.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-10532) Capacity Scheduler Auto Queue Creation: Allow auto delete queue when queue is not being used
[ https://issues.apache.org/jira/browse/YARN-10532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17283691#comment-17283691 ] Qi Zhu edited comment on YARN-10532 at 2/12/21, 1:34 PM: - !image-2021-02-12-21-32-02-267.png|width=1085,height=764! cc [~gandras] [~snemeth] [~ztang] [~epayne] I have also confirmed it in my test cluster, if you could help review the latest patch? Thanks. was (Author: zhuqi): !image-2021-02-12-21-32-02-267.png|width=1085,height=764! cc [~gandras] [~snemeth] [~ztang] [~epayne] I have also confirmed it in my test cluster, if you could review it ? Thanks. > Capacity Scheduler Auto Queue Creation: Allow auto delete queue when queue is > not being used > > > Key: YARN-10532 > URL: https://issues.apache.org/jira/browse/YARN-10532 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan >Assignee: Qi Zhu >Priority: Major > Attachments: YARN-10532.001.patch, YARN-10532.002.patch, > YARN-10532.003.patch, YARN-10532.004.patch, YARN-10532.005.patch, > YARN-10532.006.patch, YARN-10532.007.patch, YARN-10532.008.patch, > YARN-10532.009.patch, YARN-10532.010.patch, YARN-10532.011.patch, > YARN-10532.012.patch, YARN-10532.013.patch, YARN-10532.014.patch, > YARN-10532.015.patch, YARN-10532.016.patch, YARN-10532.017.patch, > YARN-10532.018.patch, YARN-10532.019.patch, YARN-10532.020.patch, > image-2021-02-12-21-32-02-267.png > > > It's better if we can delete auto-created queues when they are not in use for > a period of time (like 5 mins). It will be helpful when we have a large > number of auto-created queues (e.g. from 500 users), but only a small subset > of queues are actively used. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10532) Capacity Scheduler Auto Queue Creation: Allow auto delete queue when queue is not being used
[ https://issues.apache.org/jira/browse/YARN-10532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17283691#comment-17283691 ] Qi Zhu commented on YARN-10532: --- !image-2021-02-12-21-32-02-267.png|width=1085,height=764! cc [~gandras] [~snemeth] [~ztang] [~epayne] I have also confirmed it in my test cluster, if you could review it ? Thanks. > Capacity Scheduler Auto Queue Creation: Allow auto delete queue when queue is > not being used > > > Key: YARN-10532 > URL: https://issues.apache.org/jira/browse/YARN-10532 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan >Assignee: Qi Zhu >Priority: Major > Attachments: YARN-10532.001.patch, YARN-10532.002.patch, > YARN-10532.003.patch, YARN-10532.004.patch, YARN-10532.005.patch, > YARN-10532.006.patch, YARN-10532.007.patch, YARN-10532.008.patch, > YARN-10532.009.patch, YARN-10532.010.patch, YARN-10532.011.patch, > YARN-10532.012.patch, YARN-10532.013.patch, YARN-10532.014.patch, > YARN-10532.015.patch, YARN-10532.016.patch, YARN-10532.017.patch, > YARN-10532.018.patch, YARN-10532.019.patch, YARN-10532.020.patch, > image-2021-02-12-21-32-02-267.png > > > It's better if we can delete auto-created queues when they are not in use for > a period of time (like 5 mins). It will be helpful when we have a large > number of auto-created queues (e.g. from 500 users), but only a small subset > of queues are actively used. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10532) Capacity Scheduler Auto Queue Creation: Allow auto delete queue when queue is not being used
[ https://issues.apache.org/jira/browse/YARN-10532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qi Zhu updated YARN-10532: -- Attachment: image-2021-02-12-21-32-02-267.png > Capacity Scheduler Auto Queue Creation: Allow auto delete queue when queue is > not being used > > > Key: YARN-10532 > URL: https://issues.apache.org/jira/browse/YARN-10532 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan >Assignee: Qi Zhu >Priority: Major > Attachments: YARN-10532.001.patch, YARN-10532.002.patch, > YARN-10532.003.patch, YARN-10532.004.patch, YARN-10532.005.patch, > YARN-10532.006.patch, YARN-10532.007.patch, YARN-10532.008.patch, > YARN-10532.009.patch, YARN-10532.010.patch, YARN-10532.011.patch, > YARN-10532.012.patch, YARN-10532.013.patch, YARN-10532.014.patch, > YARN-10532.015.patch, YARN-10532.016.patch, YARN-10532.017.patch, > YARN-10532.018.patch, YARN-10532.019.patch, YARN-10532.020.patch, > image-2021-02-12-21-32-02-267.png > > > It's better if we can delete auto-created queues when they are not in use for > a period of time (like 5 mins). It will be helpful when we have a large > number of auto-created queues (e.g. from 500 users), but only a small subset > of queues are actively used. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-10625) FairScheduler: add global flag to disable AM-preemption
Peter Bacsko created YARN-10625: --- Summary: FairScheduler: add global flag to disable AM-preemption Key: YARN-10625 URL: https://issues.apache.org/jira/browse/YARN-10625 Project: Hadoop YARN Issue Type: Improvement Components: fairscheduler Affects Versions: 3.3.0 Reporter: Peter Bacsko Assignee: Peter Bacsko YARN-9537 added a feature to disable AM preemption on a per queue basis. This is a nice enhancement, but it's very inconvenient if the cluster has a lot of queues or queues dynamically created/deleted regularly (static queue configuration changes). It's a legitimate use-case to have AM preemption turned off completely. To make it easier, add property which acts as a global flag for this feature. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-10588) Percentage of queue and cluster is zero in WebUI
[
https://issues.apache.org/jira/browse/YARN-10588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17282988#comment-17282988
]
Bilwa S T edited comment on YARN-10588 at 2/12/21, 8:23 AM:
[~epayne]
Modifying *DominantResourceCalculator#isInvalidDivisor* to match logic of
*DominantResourceCalculator#divide* is nothing but returning true only if all
resource value is *0*. We already have a method called
*DominantResourceCalculator#isAllInvalidDivisor* which will return true only if
all resources are *zero*. I think we can just change isInvalidDivisor to
isAllInvalidDivisor. Correct me if i am wrong
was (Author: bilwast):
[~epayne]
Modifying *DominantResourceCalculator#isInvalidDivisor* to match logic of
*DominantResourceCalculator#divide* is nothing but returning true only if all
resource value is *0*. We already have a method called
*DominantResourceCalculator#isAllInvalidDivisor* which will return true only if
all resources are *zero*. I think we can just change isInvalidDivisor to
isAllInvalidDivisor.
> Percentage of queue and cluster is zero in WebUI
> -
>
> Key: YARN-10588
> URL: https://issues.apache.org/jira/browse/YARN-10588
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Bilwa S T
>Assignee: Bilwa S T
>Priority: Major
> Attachments: YARN-10588.001.patch, YARN-10588.002.patch,
> YARN-10588.003.patch
>
>
> Steps to reproduce:
> Configure below property in resource-types.xml
> {code:java}
>
> yarn.resource-types
> yarn.io/gpu
> {code}
> Submit a job
> In UI you can see % Of Queue and % Of Cluster is zero for the submitted
> application
>
> This is because in SchedulerApplicationAttempt has below check for
> calculating queueUsagePerc and clusterUsagePerc
> {code:java}
> if (!calc.isInvalidDivisor(cluster)) {
> float queueCapacityPerc = queue.getQueueInfo(false, false)
> .getCapacity();
> queueUsagePerc = calc.divide(cluster, usedResourceClone,
> Resources.multiply(cluster, queueCapacityPerc)) * 100;
> if (Float.isNaN(queueUsagePerc) || Float.isInfinite(queueUsagePerc)) {
> queueUsagePerc = 0.0f;
> }
> clusterUsagePerc =
> calc.divide(cluster, usedResourceClone, cluster) * 100;
> }
> {code}
> calc.isInvalidDivisor(cluster) always returns true as gpu resource is 0
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
