[jira] [Updated] (YARN-10528) maxAMShare should only be accepted for leaf queues, not parent queues
[
https://issues.apache.org/jira/browse/YARN-10528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shilun Fan updated YARN-10528:
--
Component/s: fairscheduler
(was: am)
Target Version/s: 3.2.3, 3.3.1, 3.4.0 (was: 3.4.0, 3.3.1, 3.2.3)
> maxAMShare should only be accepted for leaf queues, not parent queues
> -
>
> Key: YARN-10528
> URL: https://issues.apache.org/jira/browse/YARN-10528
> Project: Hadoop YARN
> Issue Type: Bug
> Components: fairscheduler
>Affects Versions: 3.4.0, 3.3.1, 3.2.3
>Reporter: Siddharth Ahuja
>Assignee: Siddharth Ahuja
>Priority: Major
> Fix For: 3.4.0, 3.3.1, 3.2.3
>
> Attachments: YARN-10528-branch-3.1.001.patch,
> YARN-10528-branch-3.2.001.patch, YARN-10528-branch-3.3.001.patch,
> YARN-10528.001.patch, YARN-10528.002.patch, maxAMShare for root.users (parent
> queue) has no effect as child queue does not inherit it.png
>
>
> Based on [Hadoop
> documentation|https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/FairScheduler.html],
> it is clear that {{maxAMShare}} property can only be used for *leaf queues*.
> This is similar to the {{reservation}} setting.
> However, existing code only ensures that the reservation setting is not
> accepted for "parent" queues (see
> https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/allocation/AllocationFileQueueParser.java#L226
> and
> https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/allocation/AllocationFileQueueParser.java#L233)
> but it is missing the checks for {{maxAMShare}}. Due to this, it is
> currently possible to have an allocation similar to below:
> {code}
>
>
>
> 1.0
> drf
> *
> *
>
> 1.0
> drf
>
>
> 1.0
> drf
> 1.0
>
>
> fair
>
>
>
>
>
>
>
>
> {code}
> where {{maxAMShare}} is 1.0f meaning, it is possible allocate 100% of the
> queue's resources for Application Masters. Notice above that root.users is a
> parent queue, however, it still gladly accepts {{maxAMShare}}. This is
> contrary to the documentation and in fact, it is very misleading because the
> child queues like root.users. actually do not inherit this setting at
> all and they still go on and use the default of 0.5 instead of 1.0, see the
> attached screenshot as an example.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10528) maxAMShare should only be accepted for leaf queues, not parent queues
[
https://issues.apache.org/jira/browse/YARN-10528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shilun Fan updated YARN-10528:
--
Component/s: am
Target Version/s: 3.2.3, 3.3.1, 3.4.0
Affects Version/s: 3.2.3
3.3.1
3.4.0
> maxAMShare should only be accepted for leaf queues, not parent queues
> -
>
> Key: YARN-10528
> URL: https://issues.apache.org/jira/browse/YARN-10528
> Project: Hadoop YARN
> Issue Type: Bug
> Components: am
>Affects Versions: 3.4.0, 3.3.1, 3.2.3
>Reporter: Siddharth Ahuja
>Assignee: Siddharth Ahuja
>Priority: Major
> Fix For: 3.4.0, 3.3.1, 3.2.3
>
> Attachments: YARN-10528-branch-3.1.001.patch,
> YARN-10528-branch-3.2.001.patch, YARN-10528-branch-3.3.001.patch,
> YARN-10528.001.patch, YARN-10528.002.patch, maxAMShare for root.users (parent
> queue) has no effect as child queue does not inherit it.png
>
>
> Based on [Hadoop
> documentation|https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/FairScheduler.html],
> it is clear that {{maxAMShare}} property can only be used for *leaf queues*.
> This is similar to the {{reservation}} setting.
> However, existing code only ensures that the reservation setting is not
> accepted for "parent" queues (see
> https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/allocation/AllocationFileQueueParser.java#L226
> and
> https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/allocation/AllocationFileQueueParser.java#L233)
> but it is missing the checks for {{maxAMShare}}. Due to this, it is
> currently possible to have an allocation similar to below:
> {code}
>
>
>
> 1.0
> drf
> *
> *
>
> 1.0
> drf
>
>
> 1.0
> drf
> 1.0
>
>
> fair
>
>
>
>
>
>
>
>
> {code}
> where {{maxAMShare}} is 1.0f meaning, it is possible allocate 100% of the
> queue's resources for Application Masters. Notice above that root.users is a
> parent queue, however, it still gladly accepts {{maxAMShare}}. This is
> contrary to the documentation and in fact, it is very misleading because the
> child queues like root.users. actually do not inherit this setting at
> all and they still go on and use the default of 0.5 instead of 1.0, see the
> attached screenshot as an example.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10528) maxAMShare should only be accepted for leaf queues, not parent queues
[
https://issues.apache.org/jira/browse/YARN-10528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Szilard Nemeth updated YARN-10528:
--
Fix Version/s: 3.2.3
3.1.5
3.3.1
3.4.0
> maxAMShare should only be accepted for leaf queues, not parent queues
> -
>
> Key: YARN-10528
> URL: https://issues.apache.org/jira/browse/YARN-10528
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Siddharth Ahuja
>Assignee: Siddharth Ahuja
>Priority: Major
> Fix For: 3.4.0, 3.3.1, 3.1.5, 3.2.3
>
> Attachments: YARN-10528-branch-3.1.001.patch,
> YARN-10528-branch-3.2.001.patch, YARN-10528-branch-3.3.001.patch,
> YARN-10528.001.patch, YARN-10528.002.patch, maxAMShare for root.users (parent
> queue) has no effect as child queue does not inherit it.png
>
>
> Based on [Hadoop
> documentation|https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/FairScheduler.html],
> it is clear that {{maxAMShare}} property can only be used for *leaf queues*.
> This is similar to the {{reservation}} setting.
> However, existing code only ensures that the reservation setting is not
> accepted for "parent" queues (see
> https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/allocation/AllocationFileQueueParser.java#L226
> and
> https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/allocation/AllocationFileQueueParser.java#L233)
> but it is missing the checks for {{maxAMShare}}. Due to this, it is
> currently possible to have an allocation similar to below:
> {code}
>
>
>
> 1.0
> drf
> *
> *
>
> 1.0
> drf
>
>
> 1.0
> drf
> 1.0
>
>
> fair
>
>
>
>
>
>
>
>
> {code}
> where {{maxAMShare}} is 1.0f meaning, it is possible allocate 100% of the
> queue's resources for Application Masters. Notice above that root.users is a
> parent queue, however, it still gladly accepts {{maxAMShare}}. This is
> contrary to the documentation and in fact, it is very misleading because the
> child queues like root.users. actually do not inherit this setting at
> all and they still go on and use the default of 0.5 instead of 1.0, see the
> attached screenshot as an example.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10528) maxAMShare should only be accepted for leaf queues, not parent queues
[
https://issues.apache.org/jira/browse/YARN-10528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siddharth Ahuja updated YARN-10528:
---
Attachment: (was: YARN-10528-branch-3.3.001.patch)
> maxAMShare should only be accepted for leaf queues, not parent queues
> -
>
> Key: YARN-10528
> URL: https://issues.apache.org/jira/browse/YARN-10528
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Siddharth Ahuja
>Assignee: Siddharth Ahuja
>Priority: Major
> Attachments: YARN-10528-branch-3.1.001.patch,
> YARN-10528-branch-3.2.001.patch, YARN-10528-branch-3.3.001.patch,
> YARN-10528.001.patch, YARN-10528.002.patch, maxAMShare for root.users (parent
> queue) has no effect as child queue does not inherit it.png
>
>
> Based on [Hadoop
> documentation|https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/FairScheduler.html],
> it is clear that {{maxAMShare}} property can only be used for *leaf queues*.
> This is similar to the {{reservation}} setting.
> However, existing code only ensures that the reservation setting is not
> accepted for "parent" queues (see
> https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/allocation/AllocationFileQueueParser.java#L226
> and
> https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/allocation/AllocationFileQueueParser.java#L233)
> but it is missing the checks for {{maxAMShare}}. Due to this, it is
> currently possible to have an allocation similar to below:
> {code}
>
>
>
> 1.0
> drf
> *
> *
>
> 1.0
> drf
>
>
> 1.0
> drf
> 1.0
>
>
> fair
>
>
>
>
>
>
>
>
> {code}
> where {{maxAMShare}} is 1.0f meaning, it is possible allocate 100% of the
> queue's resources for Application Masters. Notice above that root.users is a
> parent queue, however, it still gladly accepts {{maxAMShare}}. This is
> contrary to the documentation and in fact, it is very misleading because the
> child queues like root.users. actually do not inherit this setting at
> all and they still go on and use the default of 0.5 instead of 1.0, see the
> attached screenshot as an example.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10528) maxAMShare should only be accepted for leaf queues, not parent queues
[
https://issues.apache.org/jira/browse/YARN-10528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siddharth Ahuja updated YARN-10528:
---
Attachment: YARN-10528-branch-3.3.001.patch
> maxAMShare should only be accepted for leaf queues, not parent queues
> -
>
> Key: YARN-10528
> URL: https://issues.apache.org/jira/browse/YARN-10528
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Siddharth Ahuja
>Assignee: Siddharth Ahuja
>Priority: Major
> Attachments: YARN-10528-branch-3.1.001.patch,
> YARN-10528-branch-3.2.001.patch, YARN-10528-branch-3.3.001.patch,
> YARN-10528.001.patch, YARN-10528.002.patch, maxAMShare for root.users (parent
> queue) has no effect as child queue does not inherit it.png
>
>
> Based on [Hadoop
> documentation|https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/FairScheduler.html],
> it is clear that {{maxAMShare}} property can only be used for *leaf queues*.
> This is similar to the {{reservation}} setting.
> However, existing code only ensures that the reservation setting is not
> accepted for "parent" queues (see
> https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/allocation/AllocationFileQueueParser.java#L226
> and
> https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/allocation/AllocationFileQueueParser.java#L233)
> but it is missing the checks for {{maxAMShare}}. Due to this, it is
> currently possible to have an allocation similar to below:
> {code}
>
>
>
> 1.0
> drf
> *
> *
>
> 1.0
> drf
>
>
> 1.0
> drf
> 1.0
>
>
> fair
>
>
>
>
>
>
>
>
> {code}
> where {{maxAMShare}} is 1.0f meaning, it is possible allocate 100% of the
> queue's resources for Application Masters. Notice above that root.users is a
> parent queue, however, it still gladly accepts {{maxAMShare}}. This is
> contrary to the documentation and in fact, it is very misleading because the
> child queues like root.users. actually do not inherit this setting at
> all and they still go on and use the default of 0.5 instead of 1.0, see the
> attached screenshot as an example.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10528) maxAMShare should only be accepted for leaf queues, not parent queues
[
https://issues.apache.org/jira/browse/YARN-10528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siddharth Ahuja updated YARN-10528:
---
Attachment: YARN-10528-branch-3.1.001.patch
> maxAMShare should only be accepted for leaf queues, not parent queues
> -
>
> Key: YARN-10528
> URL: https://issues.apache.org/jira/browse/YARN-10528
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Siddharth Ahuja
>Assignee: Siddharth Ahuja
>Priority: Major
> Attachments: YARN-10528-branch-3.1.001.patch,
> YARN-10528-branch-3.2.001.patch, YARN-10528-branch-3.3.001.patch,
> YARN-10528.001.patch, YARN-10528.002.patch, maxAMShare for root.users (parent
> queue) has no effect as child queue does not inherit it.png
>
>
> Based on [Hadoop
> documentation|https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/FairScheduler.html],
> it is clear that {{maxAMShare}} property can only be used for *leaf queues*.
> This is similar to the {{reservation}} setting.
> However, existing code only ensures that the reservation setting is not
> accepted for "parent" queues (see
> https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/allocation/AllocationFileQueueParser.java#L226
> and
> https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/allocation/AllocationFileQueueParser.java#L233)
> but it is missing the checks for {{maxAMShare}}. Due to this, it is
> currently possible to have an allocation similar to below:
> {code}
>
>
>
> 1.0
> drf
> *
> *
>
> 1.0
> drf
>
>
> 1.0
> drf
> 1.0
>
>
> fair
>
>
>
>
>
>
>
>
> {code}
> where {{maxAMShare}} is 1.0f meaning, it is possible allocate 100% of the
> queue's resources for Application Masters. Notice above that root.users is a
> parent queue, however, it still gladly accepts {{maxAMShare}}. This is
> contrary to the documentation and in fact, it is very misleading because the
> child queues like root.users. actually do not inherit this setting at
> all and they still go on and use the default of 0.5 instead of 1.0, see the
> attached screenshot as an example.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10528) maxAMShare should only be accepted for leaf queues, not parent queues
[
https://issues.apache.org/jira/browse/YARN-10528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siddharth Ahuja updated YARN-10528:
---
Attachment: YARN-10528-branch-3.2.001.patch
> maxAMShare should only be accepted for leaf queues, not parent queues
> -
>
> Key: YARN-10528
> URL: https://issues.apache.org/jira/browse/YARN-10528
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Siddharth Ahuja
>Assignee: Siddharth Ahuja
>Priority: Major
> Attachments: YARN-10528-branch-3.2.001.patch,
> YARN-10528-branch-3.3.001.patch, YARN-10528.001.patch, YARN-10528.002.patch,
> maxAMShare for root.users (parent queue) has no effect as child queue does
> not inherit it.png
>
>
> Based on [Hadoop
> documentation|https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/FairScheduler.html],
> it is clear that {{maxAMShare}} property can only be used for *leaf queues*.
> This is similar to the {{reservation}} setting.
> However, existing code only ensures that the reservation setting is not
> accepted for "parent" queues (see
> https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/allocation/AllocationFileQueueParser.java#L226
> and
> https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/allocation/AllocationFileQueueParser.java#L233)
> but it is missing the checks for {{maxAMShare}}. Due to this, it is
> currently possible to have an allocation similar to below:
> {code}
>
>
>
> 1.0
> drf
> *
> *
>
> 1.0
> drf
>
>
> 1.0
> drf
> 1.0
>
>
> fair
>
>
>
>
>
>
>
>
> {code}
> where {{maxAMShare}} is 1.0f meaning, it is possible allocate 100% of the
> queue's resources for Application Masters. Notice above that root.users is a
> parent queue, however, it still gladly accepts {{maxAMShare}}. This is
> contrary to the documentation and in fact, it is very misleading because the
> child queues like root.users. actually do not inherit this setting at
> all and they still go on and use the default of 0.5 instead of 1.0, see the
> attached screenshot as an example.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10528) maxAMShare should only be accepted for leaf queues, not parent queues
[
https://issues.apache.org/jira/browse/YARN-10528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siddharth Ahuja updated YARN-10528:
---
Attachment: YARN-10528-branch-3.3.001.patch
> maxAMShare should only be accepted for leaf queues, not parent queues
> -
>
> Key: YARN-10528
> URL: https://issues.apache.org/jira/browse/YARN-10528
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Siddharth Ahuja
>Assignee: Siddharth Ahuja
>Priority: Major
> Attachments: YARN-10528-branch-3.3.001.patch, YARN-10528.001.patch,
> YARN-10528.002.patch, maxAMShare for root.users (parent queue) has no effect
> as child queue does not inherit it.png
>
>
> Based on [Hadoop
> documentation|https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/FairScheduler.html],
> it is clear that {{maxAMShare}} property can only be used for *leaf queues*.
> This is similar to the {{reservation}} setting.
> However, existing code only ensures that the reservation setting is not
> accepted for "parent" queues (see
> https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/allocation/AllocationFileQueueParser.java#L226
> and
> https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/allocation/AllocationFileQueueParser.java#L233)
> but it is missing the checks for {{maxAMShare}}. Due to this, it is
> currently possible to have an allocation similar to below:
> {code}
>
>
>
> 1.0
> drf
> *
> *
>
> 1.0
> drf
>
>
> 1.0
> drf
> 1.0
>
>
> fair
>
>
>
>
>
>
>
>
> {code}
> where {{maxAMShare}} is 1.0f meaning, it is possible allocate 100% of the
> queue's resources for Application Masters. Notice above that root.users is a
> parent queue, however, it still gladly accepts {{maxAMShare}}. This is
> contrary to the documentation and in fact, it is very misleading because the
> child queues like root.users. actually do not inherit this setting at
> all and they still go on and use the default of 0.5 instead of 1.0, see the
> attached screenshot as an example.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10528) maxAMShare should only be accepted for leaf queues, not parent queues
[
https://issues.apache.org/jira/browse/YARN-10528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siddharth Ahuja updated YARN-10528:
---
Attachment: YARN-10528.002.patch
> maxAMShare should only be accepted for leaf queues, not parent queues
> -
>
> Key: YARN-10528
> URL: https://issues.apache.org/jira/browse/YARN-10528
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Siddharth Ahuja
>Assignee: Siddharth Ahuja
>Priority: Major
> Attachments: YARN-10528.001.patch, YARN-10528.002.patch, maxAMShare
> for root.users (parent queue) has no effect as child queue does not inherit
> it.png
>
>
> Based on [Hadoop
> documentation|https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/FairScheduler.html],
> it is clear that {{maxAMShare}} property can only be used for *leaf queues*.
> This is similar to the {{reservation}} setting.
> However, existing code only ensures that the reservation setting is not
> accepted for "parent" queues (see
> https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/allocation/AllocationFileQueueParser.java#L226
> and
> https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/allocation/AllocationFileQueueParser.java#L233)
> but it is missing the checks for {{maxAMShare}}. Due to this, it is
> currently possible to have an allocation similar to below:
> {code}
>
>
>
> 1.0
> drf
> *
> *
>
> 1.0
> drf
>
>
> 1.0
> drf
> 1.0
>
>
> fair
>
>
>
>
>
>
>
>
> {code}
> where {{maxAMShare}} is 1.0f meaning, it is possible allocate 100% of the
> queue's resources for Application Masters. Notice above that root.users is a
> parent queue, however, it still gladly accepts {{maxAMShare}}. This is
> contrary to the documentation and in fact, it is very misleading because the
> child queues like root.users. actually do not inherit this setting at
> all and they still go on and use the default of 0.5 instead of 1.0, see the
> attached screenshot as an example.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10528) maxAMShare should only be accepted for leaf queues, not parent queues
[
https://issues.apache.org/jira/browse/YARN-10528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siddharth Ahuja updated YARN-10528:
---
Attachment: YARN-10528.001.patch
> maxAMShare should only be accepted for leaf queues, not parent queues
> -
>
> Key: YARN-10528
> URL: https://issues.apache.org/jira/browse/YARN-10528
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Siddharth Ahuja
>Assignee: Siddharth Ahuja
>Priority: Major
> Attachments: YARN-10528.001.patch, maxAMShare for root.users (parent
> queue) has no effect as child queue does not inherit it.png
>
>
> Based on [Hadoop
> documentation|https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/FairScheduler.html],
> it is clear that {{maxAMShare}} property can only be used for *leaf queues*.
> This is similar to the {{reservation}} setting.
> However, existing code only ensures that the reservation setting is not
> accepted for "parent" queues (see
> https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/allocation/AllocationFileQueueParser.java#L226
> and
> https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/allocation/AllocationFileQueueParser.java#L233)
> but it is missing the checks for {{maxAMShare}}. Due to this, it is
> currently possible to have an allocation similar to below:
> {code}
>
>
>
> 1.0
> drf
> *
> *
>
> 1.0
> drf
>
>
> 1.0
> drf
> 1.0
>
>
> fair
>
>
>
>
>
>
>
>
> {code}
> where {{maxAMShare}} is 1.0f meaning, it is possible allocate 100% of the
> queue's resources for Application Masters. Notice above that root.users is a
> parent queue, however, it still gladly accepts {{maxAMShare}}. This is
> contrary to the documentation and in fact, it is very misleading because the
> child queues like root.users. actually do not inherit this setting at
> all and they still go on and use the default of 0.5 instead of 1.0, see the
> attached screenshot as an example.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10528) maxAMShare should only be accepted for leaf queues, not parent queues
[
https://issues.apache.org/jira/browse/YARN-10528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siddharth Ahuja updated YARN-10528:
---
Attachment: (was: YARN-10528.001.patch)
> maxAMShare should only be accepted for leaf queues, not parent queues
> -
>
> Key: YARN-10528
> URL: https://issues.apache.org/jira/browse/YARN-10528
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Siddharth Ahuja
>Assignee: Siddharth Ahuja
>Priority: Major
> Attachments: maxAMShare for root.users (parent queue) has no effect
> as child queue does not inherit it.png
>
>
> Based on [Hadoop
> documentation|https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/FairScheduler.html],
> it is clear that {{maxAMShare}} property can only be used for *leaf queues*.
> This is similar to the {{reservation}} setting.
> However, existing code only ensures that the reservation setting is not
> accepted for "parent" queues (see
> https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/allocation/AllocationFileQueueParser.java#L226
> and
> https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/allocation/AllocationFileQueueParser.java#L233)
> but it is missing the checks for {{maxAMShare}}. Due to this, it is
> currently possible to have an allocation similar to below:
> {code}
>
>
>
> 1.0
> drf
> *
> *
>
> 1.0
> drf
>
>
> 1.0
> drf
> 1.0
>
>
> fair
>
>
>
>
>
>
>
>
> {code}
> where {{maxAMShare}} is 1.0f meaning, it is possible allocate 100% of the
> queue's resources for Application Masters. Notice above that root.users is a
> parent queue, however, it still gladly accepts {{maxAMShare}}. This is
> contrary to the documentation and in fact, it is very misleading because the
> child queues like root.users. actually do not inherit this setting at
> all and they still go on and use the default of 0.5 instead of 1.0, see the
> attached screenshot as an example.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10528) maxAMShare should only be accepted for leaf queues, not parent queues
[
https://issues.apache.org/jira/browse/YARN-10528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siddharth Ahuja updated YARN-10528:
---
Attachment: (was: YARN-10528.001.patch)
> maxAMShare should only be accepted for leaf queues, not parent queues
> -
>
> Key: YARN-10528
> URL: https://issues.apache.org/jira/browse/YARN-10528
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Siddharth Ahuja
>Assignee: Siddharth Ahuja
>Priority: Major
> Attachments: maxAMShare for root.users (parent queue) has no effect
> as child queue does not inherit it.png
>
>
> Based on [Hadoop
> documentation|https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/FairScheduler.html],
> it is clear that {{maxAMShare}} property can only be used for *leaf queues*.
> This is similar to the {{reservation}} setting.
> However, existing code only ensures that the reservation setting is not
> accepted for "parent" queues (see
> https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/allocation/AllocationFileQueueParser.java#L226
> and
> https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/allocation/AllocationFileQueueParser.java#L233)
> but it is missing the checks for {{maxAMShare}}. Due to this, it is
> currently possible to have an allocation similar to below:
> {code}
>
>
>
> 1.0
> drf
> *
> *
>
> 1.0
> drf
>
>
> 1.0
> drf
> 1.0
>
>
> fair
>
>
>
>
>
>
>
>
> {code}
> where {{maxAMShare}} is 1.0f meaning, it is possible allocate 100% of the
> queue's resources for Application Masters. Notice above that root.users is a
> parent queue, however, it still gladly accepts {{maxAMShare}}. This is
> contrary to the documentation and in fact, it is very misleading because the
> child queues like root.users. actually do not inherit this setting at
> all and they still go on and use the default of 0.5 instead of 1.0, see the
> attached screenshot as an example.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10528) maxAMShare should only be accepted for leaf queues, not parent queues
[
https://issues.apache.org/jira/browse/YARN-10528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siddharth Ahuja updated YARN-10528:
---
Attachment: YARN-10528.001.patch
> maxAMShare should only be accepted for leaf queues, not parent queues
> -
>
> Key: YARN-10528
> URL: https://issues.apache.org/jira/browse/YARN-10528
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Siddharth Ahuja
>Assignee: Siddharth Ahuja
>Priority: Major
> Attachments: YARN-10528.001.patch, maxAMShare for root.users (parent
> queue) has no effect as child queue does not inherit it.png
>
>
> Based on [Hadoop
> documentation|https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/FairScheduler.html],
> it is clear that {{maxAMShare}} property can only be used for *leaf queues*.
> This is similar to the {{reservation}} setting.
> However, existing code only ensures that the reservation setting is not
> accepted for "parent" queues (see
> https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/allocation/AllocationFileQueueParser.java#L226
> and
> https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/allocation/AllocationFileQueueParser.java#L233)
> but it is missing the checks for {{maxAMShare}}. Due to this, it is
> currently possible to have an allocation similar to below:
> {code}
>
>
>
> 1.0
> drf
> *
> *
>
> 1.0
> drf
>
>
> 1.0
> drf
> 1.0
>
>
> fair
>
>
>
>
>
>
>
>
> {code}
> where {{maxAMShare}} is 1.0f meaning, it is possible allocate 100% of the
> queue's resources for Application Masters. Notice above that root.users is a
> parent queue, however, it still gladly accepts {{maxAMShare}}. This is
> contrary to the documentation and in fact, it is very misleading because the
> child queues like root.users. actually do not inherit this setting at
> all and they still go on and use the default of 0.5 instead of 1.0, see the
> attached screenshot as an example.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10528) maxAMShare should only be accepted for leaf queues, not parent queues
[
https://issues.apache.org/jira/browse/YARN-10528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siddharth Ahuja updated YARN-10528:
---
Description:
Based on [Hadoop
documentation|https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/FairScheduler.html],
it is clear that {{maxAMShare}} property can only be used for *leaf queues*.
This is similar to the {{reservation}} setting.
However, existing code only ensures that the reservation setting is not
accepted for "parent" queues (see
https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/allocation/AllocationFileQueueParser.java#L226
and
https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/allocation/AllocationFileQueueParser.java#L233)
but it is missing the checks for {{maxAMShare}}. Due to this, it is currently
possible to have an allocation similar to below:
{code}
1.0
drf
*
*
1.0
drf
1.0
drf
1.0
fair
{code}
where {{maxAMShare}} is 1.0f meaning, it is possible allocate 100% of the
queue's resources for Application Masters. Notice above that root.users is a
parent queue, however, it still gladly accepts {{maxAMShare}}. This is contrary
to the documentation and in fact, it is very misleading because the child
queues like root.users. actually do not inherit this setting at all and
they still go on and use the default of 0.5 instead of 1.0, see the attached
screenshot as an example.
was:
Based on [Hadoop
documentation|https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/FairScheduler.html],
it is clear that {{maxAMShare}} property can only be used for *leaf queues*.
This is similar to the {{reservation}} setting.
However, existing code only ensures that the reservation setting is not
accepted for "parent" queues (see
https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/allocation/AllocationFileQueueParser.java#L226
and
https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/allocation/AllocationFileQueueParser.java#L233)
but it is missing the checks for {{maxAMShare}}. Due to this, it is current
possible to have an allocation similar to below:
{code}
1.0
drf
*
*
1.0
drf
1.0
drf
1.0
fair
{code}
where {{maxAMShare}} is 1.0f meaning, it is possible allocate 100% of the
queue's resources for Application Masters. Notice above that root.users is a
parent queue, however, it still gladly accepts {{maxAMShare}}. This is contrary
to the documentation and in fact, it is very misleading because the child
queues like root.users. actually do not inherit this setting at all and
they still go on and use the default of 0.5 instead of 1.0, see the attached
screenshot as an example.
> maxAMShare should only be accepted for leaf queues, not parent queues
> -
>
> Key: YARN-10528
> URL: https://issues.apache.org/jira/browse/YARN-10528
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Siddharth Ahuja
>Assignee: Siddharth Ahuja
>Priority: Major
> Attachments: maxAMShare for root.users (parent queue) has no effect
> as child queue does not inherit it.png
>
>
> Based on [Hadoop
> documentation|https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/FairScheduler.html],
> it is clear that {{maxAMShare}} property can only be used for *leaf queues*.
> This is similar to the {{reservation}} setting.
> However, existing code only ensures that the reservation setting is not
> accepted for "parent" queues (see
> https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/allocation/AllocationFileQueueParser.java#L226
> and
>
[jira] [Updated] (YARN-10528) maxAMShare should only be accepted for leaf queues, not parent queues
[
https://issues.apache.org/jira/browse/YARN-10528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siddharth Ahuja updated YARN-10528:
---
Attachment: maxAMShare for root.users (parent queue) has no effect as child
queue does not inherit it.png
> maxAMShare should only be accepted for leaf queues, not parent queues
> -
>
> Key: YARN-10528
> URL: https://issues.apache.org/jira/browse/YARN-10528
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Siddharth Ahuja
>Assignee: Siddharth Ahuja
>Priority: Major
> Attachments: maxAMShare for root.users (parent queue) has no effect
> as child queue does not inherit it.png
>
>
> Based on [Hadoop
> documentation|https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/FairScheduler.html],
> it is clear that {{maxAMShare}} property can only be used for *leaf queues*.
> This is similar to the {{reservation}} setting.
> However, existing code only ensures that the reservation setting is not
> accepted for "parent" queues (see
> https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/allocation/AllocationFileQueueParser.java#L226
> and
> https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/allocation/AllocationFileQueueParser.java#L233)
> but it is missing the checks for {{maxAMShare}}. Due to this, it is current
> possible to have an allocation similar to below:
> {code}
>
>
>
> 1.0
> drf
> *
> *
>
> 1.0
> drf
>
>
> 1.0
> drf
> 1.0
>
>
> fair
>
>
>
>
>
>
>
>
> {code}
> where {{maxAMShare}} is 1.0f meaning, it is possible allocate 100% of the
> queue's resources for Application Masters. Notice above that root.users is a
> parent queue, however, it still gladly accepts {{maxAMShare}}. This is
> contrary to the documentation and in fact, it is very misleading because the
> child queues like root.users. actually do not inherit this setting at
> all and they still go on and use the default of 0.5 instead of 1.0, see the
> attached screenshot as an example.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
