[jira] [Commented] (YARN-10505) Extend the maximum-capacity property to support Fair Scheduler migration
[
https://issues.apache.org/jira/browse/YARN-10505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17400999#comment-17400999
]
Szilard Nemeth commented on YARN-10505:
---
This jira has been split to three other jiras:
YARN-9936
YARN-10886
YARN-10887
Resolving as duplicate.
> Extend the maximum-capacity property to support Fair Scheduler migration
>
>
> Key: YARN-10505
> URL: https://issues.apache.org/jira/browse/YARN-10505
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: capacity scheduler
>Reporter: Benjamin Teke
>Assignee: Benjamin Teke
>Priority: Major
>
> Currently Fair Scheduler supports the following 3 kinds of settings:
> * Single percentage (relative to parent) i.e. "X%"
> * A set of percentages (relative to parent) i.e. "X% cpu, Y% memory"
> * Absolute resources i.e. "X mb, Y vcores"
> Please note, that the new, recommended format does not support the single
> percentage mode, only the last 2, like: “vcores=X, memory-mb=Y” or
> “vcores=X%, memory-mb=Y%” respectively.
> Tasks to accomplish:
> # It is recommended that all three formats are supported for
> maximum-capacity in CS after introducing weight mode.
> # Also we want to introduce the percentage modes relative to the cluster,
> not the parent, i.e The property root.users.maximum-capacity will mean one of
> the following things:
> ## Either Parent Percentage: maximum capacity relative to its parent. If
> it’s set to 50, then it means that the capacity is capped with respect to the
> parent. This can be covered by the current format, no change there.
> ## Or Cluster Percentage: maximum capacity expressed as a percentage of the
> overall cluster capacity. This case is the new scenario, for example:
> {{yarn.scheduler.capacity.root.users.max-capacity = c:50%}}
> {{yarn.scheduler.capacity.root.users.max-capacity = c:50%, c:30%}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10505) Extend the maximum-capacity property to support Fair Scheduler migration
[
https://issues.apache.org/jira/browse/YARN-10505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17400919#comment-17400919
]
Szilard Nemeth commented on YARN-10505:
---
Splitting this jira up.
> Extend the maximum-capacity property to support Fair Scheduler migration
>
>
> Key: YARN-10505
> URL: https://issues.apache.org/jira/browse/YARN-10505
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: capacity scheduler
>Reporter: Benjamin Teke
>Assignee: Benjamin Teke
>Priority: Major
>
> Currently Fair Scheduler supports the following 3 kinds of settings:
> * Single percentage (relative to parent) i.e. "X%"
> * A set of percentages (relative to parent) i.e. "X% cpu, Y% memory"
> * Absolute resources i.e. "X mb, Y vcores"
> Please note, that the new, recommended format does not support the single
> percentage mode, only the last 2, like: “vcores=X, memory-mb=Y” or
> “vcores=X%, memory-mb=Y%” respectively.
> Tasks to accomplish:
> # It is recommended that all three formats are supported for
> maximum-capacity in CS after introducing weight mode.
> # Also we want to introduce the percentage modes relative to the cluster,
> not the parent, i.e The property root.users.maximum-capacity will mean one of
> the following things:
> ## Either Parent Percentage: maximum capacity relative to its parent. If
> it’s set to 50, then it means that the capacity is capped with respect to the
> parent. This can be covered by the current format, no change there.
> ## Or Cluster Percentage: maximum capacity expressed as a percentage of the
> overall cluster capacity. This case is the new scenario, for example:
> {{yarn.scheduler.capacity.root.users.max-capacity = c:50%}}
> {{yarn.scheduler.capacity.root.users.max-capacity = c:50%, c:30%}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10505) Extend the maximum-capacity property to support Fair Scheduler migration
[
https://issues.apache.org/jira/browse/YARN-10505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17397407#comment-17397407
]
Eric Payne commented on YARN-10505:
---
Thanks for clearing that up!
> Extend the maximum-capacity property to support Fair Scheduler migration
>
>
> Key: YARN-10505
> URL: https://issues.apache.org/jira/browse/YARN-10505
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: capacity scheduler
>Reporter: Benjamin Teke
>Assignee: Benjamin Teke
>Priority: Major
>
> Currently Fair Scheduler supports the following 3 kinds of settings:
> * Single percentage (relative to parent) i.e. "X%"
> * A set of percentages (relative to parent) i.e. "X% cpu, Y% memory"
> * Absolute resources i.e. "X mb, Y vcores"
> Please note, that the new, recommended format does not support the single
> percentage mode, only the last 2, like: “vcores=X, memory-mb=Y” or
> “vcores=X%, memory-mb=Y%” respectively.
> Tasks to accomplish:
> # It is recommended that all three formats are supported for
> maximum-capacity in CS after introducing weight mode.
> # Also we want to introduce the percentage modes relative to the cluster,
> not the parent, i.e The property root.users.maximum-capacity will mean one of
> the following things:
> ## Either Parent Percentage: maximum capacity relative to its parent. If
> it’s set to 50, then it means that the capacity is capped with respect to the
> parent. This can be covered by the current format, no change there.
> ## Or Cluster Percentage: maximum capacity expressed as a percentage of the
> overall cluster capacity. This case is the new scenario, for example:
> {{yarn.scheduler.capacity.root.users.max-capacity = c:50%}}
> {{yarn.scheduler.capacity.root.users.max-capacity = c:50%, c:30%}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10505) Extend the maximum-capacity property to support Fair Scheduler migration
[
https://issues.apache.org/jira/browse/YARN-10505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17397403#comment-17397403
]
Benjamin Teke commented on YARN-10505:
--
[~epayne] as [~rreti] mentioned we don't plan to change the original behaviour,
so a single percentage will still mean x percent of the parent's resources.
> Extend the maximum-capacity property to support Fair Scheduler migration
>
>
> Key: YARN-10505
> URL: https://issues.apache.org/jira/browse/YARN-10505
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: capacity scheduler
>Reporter: Benjamin Teke
>Assignee: Benjamin Teke
>Priority: Major
>
> Currently Fair Scheduler supports the following 3 kinds of settings:
> * Single percentage (relative to parent) i.e. "X%"
> * A set of percentages (relative to parent) i.e. "X% cpu, Y% memory"
> * Absolute resources i.e. "X mb, Y vcores"
> Please note, that the new, recommended format does not support the single
> percentage mode, only the last 2, like: “vcores=X, memory-mb=Y” or
> “vcores=X%, memory-mb=Y%” respectively.
> Tasks to accomplish:
> # It is recommended that all three formats are supported for
> maximum-capacity in CS after introducing weight mode.
> # Also we want to introduce the percentage modes relative to the cluster,
> not the parent, i.e The property root.users.maximum-capacity will mean one of
> the following things:
> ## Either Parent Percentage: maximum capacity relative to its parent. If
> it’s set to 50, then it means that the capacity is capped with respect to the
> parent. This can be covered by the current format, no change there.
> ## Or Cluster Percentage: maximum capacity expressed as a percentage of the
> overall cluster capacity. This case is the new scenario, for example:
> {{yarn.scheduler.capacity.root.users.max-capacity = c:50%}}
> {{yarn.scheduler.capacity.root.users.max-capacity = c:50%, c:30%}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10505) Extend the maximum-capacity property to support Fair Scheduler migration
[
https://issues.apache.org/jira/browse/YARN-10505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17397393#comment-17397393
]
Rudolf Reti commented on YARN-10505:
[~epayne] You're right, we must make sure we keep BWC. Having said that, the
statement you cited refers to FS and is there to refine the scope from an
FS->CS migration point of view.
> Extend the maximum-capacity property to support Fair Scheduler migration
>
>
> Key: YARN-10505
> URL: https://issues.apache.org/jira/browse/YARN-10505
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: capacity scheduler
>Reporter: Benjamin Teke
>Assignee: Benjamin Teke
>Priority: Major
>
> Currently Fair Scheduler supports the following 3 kinds of settings:
> * Single percentage (relative to parent) i.e. "X%"
> * A set of percentages (relative to parent) i.e. "X% cpu, Y% memory"
> * Absolute resources i.e. "X mb, Y vcores"
> Please note, that the new, recommended format does not support the single
> percentage mode, only the last 2, like: “vcores=X, memory-mb=Y” or
> “vcores=X%, memory-mb=Y%” respectively.
> Tasks to accomplish:
> # It is recommended that all three formats are supported for
> maximum-capacity in CS after introducing weight mode.
> # Also we want to introduce the percentage modes relative to the cluster,
> not the parent, i.e The property root.users.maximum-capacity will mean one of
> the following things:
> ## Either Parent Percentage: maximum capacity relative to its parent. If
> it’s set to 50, then it means that the capacity is capped with respect to the
> parent. This can be covered by the current format, no change there.
> ## Or Cluster Percentage: maximum capacity expressed as a percentage of the
> overall cluster capacity. This case is the new scenario, for example:
> {{yarn.scheduler.capacity.root.users.max-capacity = c:50%}}
> {{yarn.scheduler.capacity.root.users.max-capacity = c:50%, c:30%}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10505) Extend the maximum-capacity property to support Fair Scheduler migration
[
https://issues.apache.org/jira/browse/YARN-10505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17397371#comment-17397371
]
Eric Payne commented on YARN-10505:
---
bq. Please note, that the new, recommended format does not support the single
percentage mode, only the last 2, like: “vcores=X, memory-mb=Y” or “vcores=X%,
memory-mb=Y%” respectively.
[~bteke] / [~rreti] / [~gandras],
If I understand correctly, this change would no longer support the legacy
format of specifying a single percentage for the capacity and maximum-capacity
properties. Is that correct? If so, I would not be able to support that change
because it would require users who are upgrading to make changes to their queue
configs. Logistically, from the user's perspective, this would be cumbersome. I
think it is important to keep the single percentage to support backwards
compatibility.
> Extend the maximum-capacity property to support Fair Scheduler migration
>
>
> Key: YARN-10505
> URL: https://issues.apache.org/jira/browse/YARN-10505
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: capacity scheduler
>Reporter: Benjamin Teke
>Assignee: Benjamin Teke
>Priority: Major
>
> Currently Fair Scheduler supports the following 3 kinds of settings:
> * Single percentage (relative to parent) i.e. "X%"
> * A set of percentages (relative to parent) i.e. "X% cpu, Y% memory"
> * Absolute resources i.e. "X mb, Y vcores"
> Please note, that the new, recommended format does not support the single
> percentage mode, only the last 2, like: “vcores=X, memory-mb=Y” or
> “vcores=X%, memory-mb=Y%” respectively.
> Tasks to accomplish:
> # It is recommended that all three formats are supported for
> maximum-capacity in CS after introducing weight mode.
> # Also we want to introduce the percentage modes relative to the cluster,
> not the parent, i.e The property root.users.maximum-capacity will mean one of
> the following things:
> ## Either Parent Percentage: maximum capacity relative to its parent. If
> it’s set to 50, then it means that the capacity is capped with respect to the
> parent. This can be covered by the current format, no change there.
> ## Or Cluster Percentage: maximum capacity expressed as a percentage of the
> overall cluster capacity. This case is the new scenario, for example:
> {{yarn.scheduler.capacity.root.users.max-capacity = c:50%}}
> {{yarn.scheduler.capacity.root.users.max-capacity = c:50%, c:30%}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10505) Extend the maximum-capacity property to support Fair Scheduler migration
[ https://issues.apache.org/jira/browse/YARN-10505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17342657#comment-17342657 ] Benjamin Teke commented on YARN-10505: -- [~pbacsko] thanks for the suggestion, edited the description. > Extend the maximum-capacity property to support Fair Scheduler migration > > > Key: YARN-10505 > URL: https://issues.apache.org/jira/browse/YARN-10505 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Benjamin Teke >Priority: Major > > The property root.users.maximum-capacity could mean the following things: > * Parent Percentage: maximum capacity relative to its parent. If it’s set to > 50, then it means that the capacity is capped with respect to the parent. > * Cluster Percentage: maximum capacity expressed as a percentage of the > overall cluster capacity. > > Note that Fair Scheduler supports the following settings: > * Single percentage (absolute) > * Two percentages (absolute) > * Absolute resources > > It is recommended that all three formats are supported for maximum-capacity > after introducing weight mode. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10505) Extend the maximum-capacity property to support Fair Scheduler migration
[ https://issues.apache.org/jira/browse/YARN-10505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17342641#comment-17342641 ] Peter Bacsko commented on YARN-10505: - I would change the naming a bit. "Relative Percentage" -> "Parent Percentage" "Absolute Percentage" -> "Cluster Percentage" I think it's much clearer. Absolute/relative in this context it's hard to grasp or even sounds like an oxymoron. > Extend the maximum-capacity property to support Fair Scheduler migration > > > Key: YARN-10505 > URL: https://issues.apache.org/jira/browse/YARN-10505 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Benjamin Teke >Priority: Major > > The property root.users.maximum-capacity could mean the following things: > * Relative Percentage: maximum capacity relative to its parent. If it’s set > to 50, then it means that the capacity is capped with respect to the parent. > * Absolute Percentage: maximum capacity expressed as a percentage of the > overall cluster capacity. > > Note that Fair Scheduler supports the following settings: > * Single percentage (absolute) > * Two percentages (absolute) > * Absolute resources > > It is recommended that all three formats are supported for maximum-capacity > after introducing weight mode. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
