[
https://issues.apache.org/jira/browse/YARN-11014?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Szilard Nemeth updated YARN-11014:
----------------------------------
Description:
The YARN validation API
(http://<host>:8088/ws/v1/cluster/scheduler-conf/validate) allows a
configuration in which a child queue has a greater maximum capacity value than
its parent (the same update fails when it is attempted for real) using the
DominantResourceCalculator.
For example, the following passes validation when the root queue's maximum
capacity is less than the 100000000 it is attempting to set here:
{code:java}
{
"update-queue": [
{
"queue-name": "root.test",
"params": {
"entry": [
{
"key": "maximum-capacity",
"value": "[memory=100000000,vcores=3]"
},
{
"key": "capacity",
"value": "[memory=3020,vcores=1]"
}
]
}
}
]
}
{code}
The reason for this is the newly created CapacityScheduler instance doesn't
have the active nodes of the cluster in the nodeTracker object, hence the
clusterResources will be Resources.none() during the init/reinit phase of this
new instance. If the dominantRC has invalid denominator (clusterResource being
0) it simply compares the two values resource-by-resource:
{code:java}
@Override
public int compare(Resource clusterResource, Resource lhs, Resource rhs,
boolean singleType) {
if (lhs.equals(rhs)) {
return 0;
}
if (isAllInvalidDivisor(clusterResource)) {
return this.compare(lhs, rhs);
}
...
/**
* Compare two resources - if the value for every resource type for the lhs
* is greater than that of the rhs, return 1. If the value for every resource
* type in the lhs is less than the rhs, return -1. Otherwise, return 0
*
* @param lhs resource to be compared
* @param rhs resource to be compared
* @return 0, 1, or -1
*/
private int compare(Resource lhs, Resource rhs) {
{code}
was:
The YARN validation API
(http://<host>:8088/ws/v1/cluster/scheduler-conf/validate) allows a
configuration in which a child queue has a greater maximum capacity value than
its parent (the same update fails when it is attempted for real) using the
DominantResourceCalculator. For example, the following passes validation when
the root queue's maximum capacity is less than the 100000000 it is attempting
to set here:
{code:java}
{
"update-queue": [
{
"queue-name": "root.test",
"params": {
"entry": [
{
"key": "maximum-capacity",
"value": "[memory=100000000,vcores=3]"
},
{
"key": "capacity",
"value": "[memory=3020,vcores=1]"
}
]
}
}
]
}
{code}
The reason for this is the newly created CapacityScheduler instance doesn't
have the active nodes of the cluster in the nodeTracker object, hence the
clusterResources will be Resources.none() during the init/reinit phase of this
new instance. If the dominantRC has invalid denominator (clusterResource being
0) it simply compares the two values resource-by-resource:
{code:java}
@Override
public int compare(Resource clusterResource, Resource lhs, Resource rhs,
boolean singleType) {
if (lhs.equals(rhs)) {
return 0;
}
if (isAllInvalidDivisor(clusterResource)) {
return this.compare(lhs, rhs);
}
...
/**
* Compare two resources - if the value for every resource type for the lhs
* is greater than that of the rhs, return 1. If the value for every resource
* type in the lhs is less than the rhs, return -1. Otherwise, return 0
*
* @param lhs resource to be compared
* @param rhs resource to be compared
* @return 0, 1, or -1
*/
private int compare(Resource lhs, Resource rhs) {
{code}
> YARN incorrectly validates maximum capacity resources on the validation API
> -----------------------------------------------------------------------------
>
> Key: YARN-11014
> URL: https://issues.apache.org/jira/browse/YARN-11014
> Project: Hadoop YARN
> Issue Type: Bug
> Reporter: Benjamin Teke
> Assignee: Benjamin Teke
> Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
> Time Spent: 2h 50m
> Remaining Estimate: 0h
>
> The YARN validation API
> (http://<host>:8088/ws/v1/cluster/scheduler-conf/validate) allows a
> configuration in which a child queue has a greater maximum capacity value
> than its parent (the same update fails when it is attempted for real) using
> the DominantResourceCalculator.
> For example, the following passes validation when the root queue's maximum
> capacity is less than the 100000000 it is attempting to set here:
> {code:java}
> {
> "update-queue": [
> {
> "queue-name": "root.test",
> "params": {
> "entry": [
> {
> "key": "maximum-capacity",
> "value": "[memory=100000000,vcores=3]"
> },
> {
> "key": "capacity",
> "value": "[memory=3020,vcores=1]"
> }
> ]
> }
> }
> ]
> }
> {code}
> The reason for this is the newly created CapacityScheduler instance doesn't
> have the active nodes of the cluster in the nodeTracker object, hence the
> clusterResources will be Resources.none() during the init/reinit phase of
> this new instance. If the dominantRC has invalid denominator (clusterResource
> being 0) it simply compares the two values resource-by-resource:
> {code:java}
> @Override
> public int compare(Resource clusterResource, Resource lhs, Resource rhs,
> boolean singleType) {
> if (lhs.equals(rhs)) {
> return 0;
> }
> if (isAllInvalidDivisor(clusterResource)) {
> return this.compare(lhs, rhs);
> }
> ...
> /**
> * Compare two resources - if the value for every resource type for the lhs
> * is greater than that of the rhs, return 1. If the value for every
> resource
> * type in the lhs is less than the rhs, return -1. Otherwise, return 0
> *
> * @param lhs resource to be compared
> * @param rhs resource to be compared
> * @return 0, 1, or -1
> */
> private int compare(Resource lhs, Resource rhs) {
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
