[jira] [Commented] (YARN-11014) YARN incorrectly validates maximum capacity resources on the validation API
[
https://issues.apache.org/jira/browse/YARN-11014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17523719#comment-17523719
]
Steve Loughran commented on YARN-11014:
---
FIxed in 3.3.3; updating fix versions as appropriate
> YARN incorrectly validates maximum capacity resources on the validation API
> -
>
> Key: YARN-11014
> URL: https://issues.apache.org/jira/browse/YARN-11014
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Benjamin Teke
>Assignee: Benjamin Teke
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.2.4, 3.3.3
>
> Time Spent: 3h 10m
> Remaining Estimate: 0h
>
> The YARN validation API
> (http://:8088/ws/v1/cluster/scheduler-conf/validate) allows a
> configuration in which a child queue has a greater maximum capacity value
> than its parent (the same update fails when it is attempted for real) using
> the DominantResourceCalculator.
> For example, the following passes validation when the root queue's maximum
> capacity is less than the 1 it is attempting to set here:
> {code:java}
> {
> "update-queue": [
> {
> "queue-name": "root.test",
> "params": {
> "entry": [
> {
> "key": "maximum-capacity",
> "value": "[memory=1,vcores=3]"
> },
> {
> "key": "capacity",
> "value": "[memory=3020,vcores=1]"
> }
> ]
> }
> }
> ]
> }
> {code}
> The reason for this is the newly created CapacityScheduler instance doesn't
> have the active nodes of the cluster in the nodeTracker object, hence the
> clusterResources will be Resources.none() during the init/reinit phase of
> this new instance.
> If the dominantRC has invalid denominator (clusterResource being 0) it simply
> compares the two values resource-by-resource:
> {code:java}
> @Override
> public int compare(Resource clusterResource, Resource lhs, Resource rhs,
> boolean singleType) {
> if (lhs.equals(rhs)) {
> return 0;
> }
> if (isAllInvalidDivisor(clusterResource)) {
> return this.compare(lhs, rhs);
> }
> ...
> /**
>* Compare two resources - if the value for every resource type for the lhs
>* is greater than that of the rhs, return 1. If the value for every
> resource
>* type in the lhs is less than the rhs, return -1. Otherwise, return 0
>*
>* @param lhs resource to be compared
>* @param rhs resource to be compared
>* @return 0, 1, or -1
>*/
>private int compare(Resource lhs, Resource rhs) {
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11014) YARN incorrectly validates maximum capacity resources on the validation API
[
https://issues.apache.org/jira/browse/YARN-11014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17456378#comment-17456378
]
Szilard Nemeth commented on YARN-11014:
---
Reopening as backport PRs are still in progress.
> YARN incorrectly validates maximum capacity resources on the validation API
> -
>
> Key: YARN-11014
> URL: https://issues.apache.org/jira/browse/YARN-11014
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Benjamin Teke
>Assignee: Benjamin Teke
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
> Time Spent: 2.5h
> Remaining Estimate: 0h
>
> The YARN validation API
> (http://:8088/ws/v1/cluster/scheduler-conf/validate) allows a
> configuration in which a child queue has a greater maximum capacity value
> than its parent (the same update fails when it is attempted for real) using
> the DominantResourceCalculator. For example, the following passes validation
> when the root queue's maximum capacity is less than the 1 it is
> attempting to set here:
> {code:java}
> {
> "update-queue": [
> {
> "queue-name": "root.test",
> "params": {
> "entry": [
> {
> "key": "maximum-capacity",
> "value": "[memory=1,vcores=3]"
> },
> {
> "key": "capacity",
> "value": "[memory=3020,vcores=1]"
> }
> ]
> }
> }
> ]
> }
> {code}
> The reason for this is the newly created CapacityScheduler instance doesn't
> have the active nodes of the cluster in the nodeTracker object, hence the
> clusterResources will be Resources.none() during the init/reinit phase of
> this new instance. If the dominantRC has invalid denominator (clusterResource
> being 0) it simply compares the two values resource-by-resource:
> {code:java}
> @Override
> public int compare(Resource clusterResource, Resource lhs, Resource rhs,
> boolean singleType) {
> if (lhs.equals(rhs)) {
> return 0;
> }
> if (isAllInvalidDivisor(clusterResource)) {
> return this.compare(lhs, rhs);
> }
> ...
> /**
>* Compare two resources - if the value for every resource type for the lhs
>* is greater than that of the rhs, return 1. If the value for every
> resource
>* type in the lhs is less than the rhs, return -1. Otherwise, return 0
>*
>* @param lhs resource to be compared
>* @param rhs resource to be compared
>* @return 0, 1, or -1
>*/
>private int compare(Resource lhs, Resource rhs) {
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
