[jira] [Commented] (YARN-9593) Updating scheduler conf with comma in config value fails

2019-06-10 Thread Anthony Hsu (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-9593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16860396#comment-16860396
 ] 

Anthony Hsu commented on YARN-9593:
---

Thanks, [~jhung]. I don't have bandwidth at the moment, but glad we're agreed 
on the approach. I think this would be a good starter task.

> Updating scheduler conf with comma in config value fails
> 
>
> Key: YARN-9593
> URL: https://issues.apache.org/jira/browse/YARN-9593
> Project: Hadoop YARN
>  Issue Type: Bug
>Affects Versions: 2.9.0, 3.0.0, 3.2.0, 3.1.2
>Reporter: Anthony Hsu
>Priority: Major
>
> For example:
> {code:java}
> $ yarn schedulerconf -update "root.gridops:acl_administer_queue=user1,user2 
> group1,group2"
> Specify configuration key value as confKey=confVal.{code}
> This fails because there is a comma in the config value and the SchedConfCLI 
> splits on comma first, expecting each split to a k=v pair.
> {noformat}
> void globalUpdates(String args, SchedConfUpdateInfo updateInfo) {
>   if (args == null) {
> return;
>   }
>   HashMap globalUpdates = new HashMap<>();
>   for (String globalUpdate : args.split(",")) {
> putKeyValuePair(globalUpdates, globalUpdate);
>   }
>   updateInfo.setGlobalParams(globalUpdates);
> }{noformat}
> Cc: [~jhung]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Comment Edited] (YARN-3213) Respect labels in Capacity Scheduler when computing user-limit

2019-06-05 Thread Anthony Hsu (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-3213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16857180#comment-16857180
 ] 

Anthony Hsu edited comment on YARN-3213 at 6/6/19 12:37 AM:


Thanks for confirming, Wangda. Created YARN-9603 for updating CS to support 
specifying different minimum-user-limit-percent and user-limit-factor on a per 
node label basis.


was (Author: erwaman):
Thanks for confirming, Wangda. Will file a Jira for updating CS to support 
specifying different minimum-user-limit-percent and user-limit-factor on a per 
node label basis.

> Respect labels in Capacity Scheduler when computing user-limit
> --
>
> Key: YARN-3213
> URL: https://issues.apache.org/jira/browse/YARN-3213
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: capacityscheduler
>Reporter: Wangda Tan
>Assignee: Wangda Tan
>Priority: Major
>
> Now we can support node-labels in Capacity Scheduler, but user-limit 
> computing doesn't respect node-labels enough, we should fix that.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Created] (YARN-9603) Support setting user limit percent and user limit factor for a queue on a per node label basis

2019-06-05 Thread Anthony Hsu (JIRA)
Anthony Hsu created YARN-9603:
-

 Summary: Support setting user limit percent and user limit factor 
for a queue on a per node label basis
 Key: YARN-9603
 URL: https://issues.apache.org/jira/browse/YARN-9603
 Project: Hadoop YARN
  Issue Type: New Feature
Reporter: Anthony Hsu


User limit settings (minimum-user-limit-percent, user-limit-factor) for queues 
can NOT be set on a per node label basis. However, capacity settings (capacity, 
maximum-capacity) CAN be set on a per node label basis. This Jira is for 
supporting setting different user limits per queue per node label. For more 
context, see discussion in YARN-3213.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-3213) Respect labels in Capacity Scheduler when computing user-limit

2019-06-05 Thread Anthony Hsu (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-3213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16857180#comment-16857180
 ] 

Anthony Hsu commented on YARN-3213:
---

Thanks for confirming, Wangda. Will file a Jira for updating CS to support 
specifying different minimum-user-limit-percent and user-limit-factor on a per 
node label basis.

> Respect labels in Capacity Scheduler when computing user-limit
> --
>
> Key: YARN-3213
> URL: https://issues.apache.org/jira/browse/YARN-3213
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: capacityscheduler
>Reporter: Wangda Tan
>Assignee: Wangda Tan
>Priority: Major
>
> Now we can support node-labels in Capacity Scheduler, but user-limit 
> computing doesn't respect node-labels enough, we should fix that.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-3213) Respect labels in Capacity Scheduler when computing user-limit

2019-06-05 Thread Anthony Hsu (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-3213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16857020#comment-16857020
 ] 

Anthony Hsu commented on YARN-3213:
---

Hi [~leftnoteasy],

It seems even after YARN-3361, you are unable to set user-limit for a queue on 
a per-label basis. In CapacitySchedulerConfiguration, I see

* getLabeledQueueCapacity / setCapacityByLabel
* getLabeledQueueMaximumCapacity / setMaximumCapacityByLabel

but I don't see corresponding per-label methods for user limit and user limit 
factor. Is this intentional?

> Respect labels in Capacity Scheduler when computing user-limit
> --
>
> Key: YARN-3213
> URL: https://issues.apache.org/jira/browse/YARN-3213
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: capacityscheduler
>Reporter: Wangda Tan
>Assignee: Wangda Tan
>Priority: Major
>
> Now we can support node-labels in Capacity Scheduler, but user-limit 
> computing doesn't respect node-labels enough, we should fix that.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-9593) Updating scheduler conf with comma in config value fails

2019-05-31 Thread Anthony Hsu (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-9593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16853546#comment-16853546
 ] 

Anthony Hsu commented on YARN-9593:
---

Hi [~jhung],

Thanks for sharing the workaround. That works for me for now.

I think the CLI can be updated to ignore escaped commas ({{\,}}) to support 
confs with commas (which can escape the commas). How does that sound to you?

> Updating scheduler conf with comma in config value fails
> 
>
> Key: YARN-9593
> URL: https://issues.apache.org/jira/browse/YARN-9593
> Project: Hadoop YARN
>  Issue Type: Bug
>Affects Versions: 2.9.0, 3.0.0, 3.2.0, 3.1.2
>Reporter: Anthony Hsu
>Priority: Major
>
> For example:
> {code:java}
> $ yarn schedulerconf -update "root.gridops:acl_administer_queue=user1,user2 
> group1,group2"
> Specify configuration key value as confKey=confVal.{code}
> This fails because there is a comma in the config value and the SchedConfCLI 
> splits on comma first, expecting each split to a k=v pair.
> {noformat}
> void globalUpdates(String args, SchedConfUpdateInfo updateInfo) {
>   if (args == null) {
> return;
>   }
>   HashMap globalUpdates = new HashMap<>();
>   for (String globalUpdate : args.split(",")) {
> putKeyValuePair(globalUpdates, globalUpdate);
>   }
>   updateInfo.setGlobalParams(globalUpdates);
> }{noformat}
> Cc: [~jhung]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Created] (YARN-9593) Updating scheduler conf with comma in config value fails

2019-05-31 Thread Anthony Hsu (JIRA)
Anthony Hsu created YARN-9593:
-

 Summary: Updating scheduler conf with comma in config value fails
 Key: YARN-9593
 URL: https://issues.apache.org/jira/browse/YARN-9593
 Project: Hadoop YARN
  Issue Type: Bug
Affects Versions: 3.1.2, 3.2.0, 3.0.0, 2.9.0
Reporter: Anthony Hsu


For example:
{code:java}
$ yarn schedulerconf -update "root.gridops:acl_administer_queue=user1,user2 
group1,group2"

Specify configuration key value as confKey=confVal.{code}
This fails because there is a comma in the config value and the SchedConfCLI 
splits on comma first, expecting each split to a k=v pair.
{noformat}
void globalUpdates(String args, SchedConfUpdateInfo updateInfo) {
  if (args == null) {
return;
  }
  HashMap globalUpdates = new HashMap<>();
  for (String globalUpdate : args.split(",")) {
putKeyValuePair(globalUpdates, globalUpdate);
  }
  updateInfo.setGlobalParams(globalUpdates);
}{noformat}
Cc: [~jhung]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-9559) Create AbstractContainersLauncher for pluggable ContainersLauncher logic

2019-05-16 Thread Anthony Hsu (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-9559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16841494#comment-16841494
 ] 

Anthony Hsu commented on YARN-9559:
---

+1 on your patch (non-binding).

> Create AbstractContainersLauncher for pluggable ContainersLauncher logic
> 
>
> Key: YARN-9559
> URL: https://issues.apache.org/jira/browse/YARN-9559
> Project: Hadoop YARN
>  Issue Type: Task
>Reporter: Jonathan Hung
>Assignee: Jonathan Hung
>Priority: Major
> Attachments: YARN-9559.001.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-9529) Log correct cpu controller path on error

2019-05-03 Thread Anthony Hsu (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-9529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16832958#comment-16832958
 ] 

Anthony Hsu commented on YARN-9529:
---

+1 (non-binding). What were you trying to debug that requires this change?

> Log correct cpu controller path on error
> 
>
> Key: YARN-9529
> URL: https://issues.apache.org/jira/browse/YARN-9529
> Project: Hadoop YARN
>  Issue Type: Improvement
>Reporter: Jonathan Hung
>Assignee: Jonathan Hung
>Priority: Major
> Attachments: YARN-9529.001.patch
>
>
> The base cpu controller path is logged instead of the hadoop cgroup path.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-9412) Backport YARN-6909 to branch-2

2019-03-27 Thread Anthony Hsu (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-9412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16803465#comment-16803465
 ] 

Anthony Hsu commented on YARN-9412:
---

Awesome!

> Backport YARN-6909 to branch-2
> --
>
> Key: YARN-9412
> URL: https://issues.apache.org/jira/browse/YARN-9412
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Jonathan Hung
>Assignee: Jonathan Hung
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Comment Edited] (YARN-9409) Port resource type changes from YARN-7237 to branch-3.0/branch-2

2019-03-26 Thread Anthony Hsu (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-9409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16802136#comment-16802136
 ] 

Anthony Hsu edited comment on YARN-9409 at 3/26/19 8:27 PM:


+1 (non-binding).

Can you also backport YARN-6909 which is needed to write end-to-end unit tests 
that involve request new resource types like GPUs?

 It probably makes sense to do this in a separate subtask of YARN-8200. I'd be 
happy to review the patch.


was (Author: erwaman):
+1 (non-binding).

Can you also backport YARN-6909 which is needed to write end-to-end unit tests 
that involve request new resource types like GPUs?

 

> Port resource type changes from YARN-7237 to branch-3.0/branch-2
> 
>
> Key: YARN-9409
> URL: https://issues.apache.org/jira/browse/YARN-9409
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Jonathan Hung
>Assignee: Jonathan Hung
>Priority: Major
> Attachments: YARN-9409-YARN-8200.branch3.001.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Comment Edited] (YARN-9409) Port resource type changes from YARN-7237 to branch-3.0/branch-2

2019-03-26 Thread Anthony Hsu (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-9409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16802136#comment-16802136
 ] 

Anthony Hsu edited comment on YARN-9409 at 3/26/19 8:27 PM:


+1 (non-binding).

Can you also backport YARN-6909 which is needed to write end-to-end unit tests 
that involve request new resource types like GPUs?

It probably makes sense to do this in a separate subtask of YARN-8200. I'd be 
happy to review the patch.


was (Author: erwaman):
+1 (non-binding).

Can you also backport YARN-6909 which is needed to write end-to-end unit tests 
that involve request new resource types like GPUs?

 It probably makes sense to do this in a separate subtask of YARN-8200. I'd be 
happy to review the patch.

> Port resource type changes from YARN-7237 to branch-3.0/branch-2
> 
>
> Key: YARN-9409
> URL: https://issues.apache.org/jira/browse/YARN-9409
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Jonathan Hung
>Assignee: Jonathan Hung
>Priority: Major
> Attachments: YARN-9409-YARN-8200.branch3.001.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-9409) Port resource type changes from YARN-7237 to branch-3.0/branch-2

2019-03-26 Thread Anthony Hsu (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-9409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16802136#comment-16802136
 ] 

Anthony Hsu commented on YARN-9409:
---

+1 (non-binding).

Can you also backport YARN-6909 which is needed to write end-to-end unit tests 
that involve request new resource types like GPUs?

 

> Port resource type changes from YARN-7237 to branch-3.0/branch-2
> 
>
> Key: YARN-9409
> URL: https://issues.apache.org/jira/browse/YARN-9409
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Jonathan Hung
>Assignee: Jonathan Hung
>Priority: Major
> Attachments: YARN-9409-YARN-8200.branch3.001.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-9272) Backport YARN-7738 for refreshing max allocation for multiple resource types

2019-03-26 Thread Anthony Hsu (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-9272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16802133#comment-16802133
 ] 

Anthony Hsu commented on YARN-9272:
---

I looked at the branch-2 patch. It looks good. +1 (non-binding).

> Backport YARN-7738 for refreshing max allocation for multiple resource types
> 
>
> Key: YARN-9272
> URL: https://issues.apache.org/jira/browse/YARN-9272
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Jonathan Hung
>Assignee: Jonathan Hung
>Priority: Major
> Attachments: YARN-9272-YARN-8200.001.patch, 
> YARN-9272-YARN-8200.branch3.001.patch, YARN-9272-YARN-8200.branch3.002.patch
>
>
> Need to port to YARN-8200.branch3 (for branch-3.0) and YARN-8200 (for 
> branch-2)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-6686) Support for adding and removing queue mappings

2019-02-14 Thread Anthony Hsu (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-6686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16768763#comment-16768763
 ] 

Anthony Hsu commented on YARN-6686:
---

Hi [~csexz],

I like your proposal of using  and . I have one suggestion. Instead 
of having separate  and , I think everything should be 
an  ( to me sounds like it's a read-only request, not 
an update request).
 * If  just contains text, then it's a complete replacement.
 * If  has  and  children, then it's an addition, removal, or 
update

||Operation|| Description/Notes||
|Addition|(empty)|u::,u::,...|Adds the mappings 
u::,u::,... |
|Removal|u::,u::,...|(empty)|Removes the mappings 
u::,u::,...|
|Update|u::,u::,...|u::,u::,...|Updates
 the mappings for users , , ... from , , ... to 
, , ...
 
The  and  should contain the same number of items.|

> Support for adding and removing queue mappings
> --
>
> Key: YARN-6686
> URL: https://issues.apache.org/jira/browse/YARN-6686
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Jonathan Hung
>Assignee: Jonathan Hung
>Priority: Major
> Attachments: YARN-6686-YARN-5734.001.patch
>
>
> Right now capacity scheduler uses UserGroupMappingPlacementRule to determine 
> queue mappings. This rule stores mappings in 
> {{yarn.scheduler.capacity.queue-mappings}}. For users with a large number of 
> mappings, adding or removing queue mappings becomes infeasible.
> Need to come up with a way to add/remove individual mappings, for any/all 
> different configured placement rules.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-9174) branch-3.0/branch-2 refactoring of GpuDevice class

2019-01-07 Thread Anthony Hsu (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-9174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16736133#comment-16736133
 ] 

Anthony Hsu commented on YARN-9174:
---

What is the motivation for backporting the GpuDevice class refactoring to 
branch-2? Is there another Jira that depends on this?

> branch-3.0/branch-2 refactoring of GpuDevice class
> --
>
> Key: YARN-9174
> URL: https://issues.apache.org/jira/browse/YARN-9174
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Jonathan Hung
>Assignee: Jonathan Hung
>Priority: Major
> Attachments: YARN-9174-YARN-8200.001.patch, 
> YARN-9174-YARN-8200.branch3.001.patch, YARN-9174-YARN-8200.branch3.002.patch
>
>
> YARN-7224 does two main things:
>  # refactors Gpu device numbers to a separate GpuDevice class,
>  # adds Docker support for Gpus
> This ticket is for doing *only* the GpuDevice class refactoring so we have 
> this logic in branch-3.0 and branch-2



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-9085) Guaranteed and MaxCapacity CSQueueMetrics

2018-12-07 Thread Anthony Hsu (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-9085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16713098#comment-16713098
 ] 

Anthony Hsu commented on YARN-9085:
---

Latest patch looks good to me. +1 (non-binding)

> Guaranteed and MaxCapacity CSQueueMetrics
> -
>
> Key: YARN-9085
> URL: https://issues.apache.org/jira/browse/YARN-9085
> Project: Hadoop YARN
>  Issue Type: Improvement
>Affects Versions: 2.9.3
>Reporter: Jonathan Hung
>Assignee: Jonathan Hung
>Priority: Major
> Attachments: YARN-9085.001.patch, YARN-9085.002.patch
>
>
> Would be useful to have Absolute Capacity/Absolute Max Capacity for queues to 
> compare against allocated/pending/etc.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-9085) Guaranteed and MaxCapacity CSQueueMetrics

2018-12-07 Thread Anthony Hsu (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-9085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16713094#comment-16713094
 ] 

Anthony Hsu commented on YARN-9085:
---

{quote}GPU metrics should be addressable in YARN-8842 (but this is currently 
only in 3.3.0)
{quote}
Can we backport this to branch 2.x? GPU metrics are important for those doing 
deep learning on GPUs.

> Guaranteed and MaxCapacity CSQueueMetrics
> -
>
> Key: YARN-9085
> URL: https://issues.apache.org/jira/browse/YARN-9085
> Project: Hadoop YARN
>  Issue Type: Improvement
>Affects Versions: 2.9.3
>Reporter: Jonathan Hung
>Assignee: Jonathan Hung
>Priority: Major
> Attachments: YARN-9085.001.patch, YARN-9085.002.patch
>
>
> Would be useful to have Absolute Capacity/Absolute Max Capacity for queues to 
> compare against allocated/pending/etc.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-9085) Guaranteed and MaxCapacity CSQueueMetrics

2018-12-06 Thread Anthony Hsu (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-9085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16711709#comment-16711709
 ] 

Anthony Hsu commented on YARN-9085:
---

Looks good overall. A few questions/comments:
 * In CSQueueMetrics.java, I see the comment

{code:java}
//Metrics updated only for "default" partition{code}
How come metrics are not updated for non-default partitions? Are any metrics 
available for non-default partitions?
 * What about GPU metrics? Can CSQueueMetrics collect those, too?
 * Regarding
{noformat}
 if (nodePartition == null) {
   for (String partition : 
Sets.union(queueCapacities.getNodePartitionsSet(),
   queueResourceUsage.getNodePartitionsSet())) {
-updateUsedCapacity(rc, nlm.getResourceByLabel(partition, cluster),
-partition, childQueue);
+updateUsedCapacity(rc, partitionResource, partition, childQueue);
   }
+  updateConfiguredCapacities(rc, partitionResource, childQueue);
 } else {
-  updateUsedCapacity(rc, nlm.getResourceByLabel(nodePartition, cluster),
-  nodePartition, childQueue);
+  updateUsedCapacity(rc, partitionResource, nodePartition, childQueue);
 }
{noformat}
Seems to me the *updateConfiguredCapacities* call you added should be inside 
the for loop and should also take in a *partition* parameter like the 
*updateUsedCapacity* call does. In the future, metrics may be collected for 
non-default partitions as well.
Also, I think the *else* block should also have a *updateConfiguredCapacities* 
call (in case in the future, we collect non-default partition metrics, too.

> Guaranteed and MaxCapacity CSQueueMetrics
> -
>
> Key: YARN-9085
> URL: https://issues.apache.org/jira/browse/YARN-9085
> Project: Hadoop YARN
>  Issue Type: Improvement
>Affects Versions: 2.9.3
>Reporter: Jonathan Hung
>Assignee: Jonathan Hung
>Priority: Major
> Attachments: YARN-9085.001.patch
>
>
> Would be useful to have Absolute Capacity/Absolute Max Capacity for queues to 
> compare against allocated/pending/etc.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org