[jira] [Commented] (YARN-6299) FairSharePolicy is incorrect when demand is less than min share
[
https://issues.apache.org/jira/browse/YARN-6299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16071998#comment-16071998
]
zhengchenyu commented on YARN-6299:
---
In FSAppAttempt, getMinShare return Resources.none(). So demand couldn't be
less than min share on app level.
> FairSharePolicy is incorrect when demand is less than min share
> ---
>
> Key: YARN-6299
> URL: https://issues.apache.org/jira/browse/YARN-6299
> Project: Hadoop YARN
> Issue Type: Bug
> Components: fairscheduler
>Affects Versions: 3.0.0-alpha2
>Reporter: Daniel Templeton
>
> {code}
> Resource resourceUsage1 = s1.getResourceUsage();
> Resource resourceUsage2 = s2.getResourceUsage();
> Resource minShare1 = Resources.min(RESOURCE_CALCULATOR, null,
> s1.getMinShare(), s1.getDemand());
> Resource minShare2 = Resources.min(RESOURCE_CALCULATOR, null,
> s2.getMinShare(), s2.getDemand());
> boolean s1Needy = Resources.lessThan(RESOURCE_CALCULATOR, null,
> resourceUsage1, minShare1);
> boolean s2Needy = Resources.lessThan(RESOURCE_CALCULATOR, null,
> resourceUsage2, minShare2);
> minShareRatio1 = (double) resourceUsage1.getMemorySize()
> / Resources.max(RESOURCE_CALCULATOR, null, minShare1,
> ONE).getMemorySize();
> minShareRatio2 = (double) resourceUsage2.getMemorySize()
> / Resources.max(RESOURCE_CALCULATOR, null, minShare2,
> ONE).getMemorySize();
> {code}
> If demand is less than min share, then an app will be flagged as needy if it
> has demand that is higher than its usage, which happens any time the app has
> been assigned resources that it hasn't started using yet. That sounds wrong
> to me. [~kasha], [~yufeigu]?
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6299) FairSharePolicy is incorrect when demand is less than min share
[
https://issues.apache.org/jira/browse/YARN-6299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15927256#comment-15927256
]
Yufei Gu commented on YARN-6299:
In the case, assigning resource to the one with fewer usage seems reasonable,
and the min share ratio can guarantee that if both demands are under min share.
I've thought about another case, one demand is less than min share and the
other is greater. Assume the min share is 10G.
{code}
demand1 = 5
demand2 = 12
resourceUsage1 = 4
resourceUsage2 = 7
minShare1 = 5
minShare2 = 10
s1Needy = true (4 < 5)
s2Needy = true (7 < 10)
minshareRatio1 = 0.8
minshareRatio2 = 0.7
{code}
We prefer app2 since minshareRatio2 < minshareRatio2, but if we don't do
{{min(demand, minShare)}}, then minshareRatios will be:
{code}
minshareRatio1 = 4/10 = 0.4
minshareRatio2 = 7/10 = 0.7
{code}
Seems like we could prefer app1.
> FairSharePolicy is incorrect when demand is less than min share
> ---
>
> Key: YARN-6299
> URL: https://issues.apache.org/jira/browse/YARN-6299
> Project: Hadoop YARN
> Issue Type: Bug
> Components: fairscheduler
>Affects Versions: 3.0.0-alpha2
>Reporter: Daniel Templeton
>
> {code}
> Resource resourceUsage1 = s1.getResourceUsage();
> Resource resourceUsage2 = s2.getResourceUsage();
> Resource minShare1 = Resources.min(RESOURCE_CALCULATOR, null,
> s1.getMinShare(), s1.getDemand());
> Resource minShare2 = Resources.min(RESOURCE_CALCULATOR, null,
> s2.getMinShare(), s2.getDemand());
> boolean s1Needy = Resources.lessThan(RESOURCE_CALCULATOR, null,
> resourceUsage1, minShare1);
> boolean s2Needy = Resources.lessThan(RESOURCE_CALCULATOR, null,
> resourceUsage2, minShare2);
> minShareRatio1 = (double) resourceUsage1.getMemorySize()
> / Resources.max(RESOURCE_CALCULATOR, null, minShare1,
> ONE).getMemorySize();
> minShareRatio2 = (double) resourceUsage2.getMemorySize()
> / Resources.max(RESOURCE_CALCULATOR, null, minShare2,
> ONE).getMemorySize();
> {code}
> If demand is less than min share, then an app will be flagged as needy if it
> has demand that is higher than its usage, which happens any time the app has
> been assigned resources that it hasn't started using yet. That sounds wrong
> to me. [~kasha], [~yufeigu]?
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6299) FairSharePolicy is incorrect when demand is less than min share
[
https://issues.apache.org/jira/browse/YARN-6299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15905920#comment-15905920
]
Daniel Templeton commented on YARN-6299:
Hmmm... Thinking about it a little further, I'm not sure it actually matters
that much which one we prefer. They're both under the min share, so scheduling
either one is fine... Maybe?
> FairSharePolicy is incorrect when demand is less than min share
> ---
>
> Key: YARN-6299
> URL: https://issues.apache.org/jira/browse/YARN-6299
> Project: Hadoop YARN
> Issue Type: Bug
> Components: fairscheduler
>Affects Versions: 3.0.0-alpha2
>Reporter: Daniel Templeton
>
> {code}
> Resource resourceUsage1 = s1.getResourceUsage();
> Resource resourceUsage2 = s2.getResourceUsage();
> Resource minShare1 = Resources.min(RESOURCE_CALCULATOR, null,
> s1.getMinShare(), s1.getDemand());
> Resource minShare2 = Resources.min(RESOURCE_CALCULATOR, null,
> s2.getMinShare(), s2.getDemand());
> boolean s1Needy = Resources.lessThan(RESOURCE_CALCULATOR, null,
> resourceUsage1, minShare1);
> boolean s2Needy = Resources.lessThan(RESOURCE_CALCULATOR, null,
> resourceUsage2, minShare2);
> minShareRatio1 = (double) resourceUsage1.getMemorySize()
> / Resources.max(RESOURCE_CALCULATOR, null, minShare1,
> ONE).getMemorySize();
> minShareRatio2 = (double) resourceUsage2.getMemorySize()
> / Resources.max(RESOURCE_CALCULATOR, null, minShare2,
> ONE).getMemorySize();
> {code}
> If demand is less than min share, then an app will be flagged as needy if it
> has demand that is higher than its usage, which happens any time the app has
> been assigned resources that it hasn't started using yet. That sounds wrong
> to me. [~kasha], [~yufeigu]?
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6299) FairSharePolicy is incorrect when demand is less than min share
[
https://issues.apache.org/jira/browse/YARN-6299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15905916#comment-15905916
]
Daniel Templeton commented on YARN-6299:
My concern is that if I have two apps that are both under min share, we will
prefer the app that has used fewer of the resources assigned to it. For
example, assume min share is 16GB. Assume app1 has requested 10GB and used
4GB. app2 has also requested 10GB and used 8GB.
{code}
resourceUsage1 = 4
resourceUsage2 = 8
minShare1 = 10
minShare2 = 10
s1Needy = true
s2Needy = true
minshareRatio1 = 0.4
minshareRatio2 = 0.8
{code}
We will therefore return {{0.4 - 0.8}}, which is negative, meaning that we
prefer app1 to app2. Unless I'm misunderstanding something.
> FairSharePolicy is incorrect when demand is less than min share
> ---
>
> Key: YARN-6299
> URL: https://issues.apache.org/jira/browse/YARN-6299
> Project: Hadoop YARN
> Issue Type: Bug
> Components: fairscheduler
>Affects Versions: 3.0.0-alpha2
>Reporter: Daniel Templeton
>
> {code}
> Resource resourceUsage1 = s1.getResourceUsage();
> Resource resourceUsage2 = s2.getResourceUsage();
> Resource minShare1 = Resources.min(RESOURCE_CALCULATOR, null,
> s1.getMinShare(), s1.getDemand());
> Resource minShare2 = Resources.min(RESOURCE_CALCULATOR, null,
> s2.getMinShare(), s2.getDemand());
> boolean s1Needy = Resources.lessThan(RESOURCE_CALCULATOR, null,
> resourceUsage1, minShare1);
> boolean s2Needy = Resources.lessThan(RESOURCE_CALCULATOR, null,
> resourceUsage2, minShare2);
> minShareRatio1 = (double) resourceUsage1.getMemorySize()
> / Resources.max(RESOURCE_CALCULATOR, null, minShare1,
> ONE).getMemorySize();
> minShareRatio2 = (double) resourceUsage2.getMemorySize()
> / Resources.max(RESOURCE_CALCULATOR, null, minShare2,
> ONE).getMemorySize();
> {code}
> If demand is less than min share, then an app will be flagged as needy if it
> has demand that is higher than its usage, which happens any time the app has
> been assigned resources that it hasn't started using yet. That sounds wrong
> to me. [~kasha], [~yufeigu]?
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6299) FairSharePolicy is incorrect when demand is less than min share
[
https://issues.apache.org/jira/browse/YARN-6299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15901645#comment-15901645
]
Yufei Gu commented on YARN-6299:
It makes sense to me that an app is always needy when its demand is less than
min share. [~templedf], can you elaborate your concern?
> FairSharePolicy is incorrect when demand is less than min share
> ---
>
> Key: YARN-6299
> URL: https://issues.apache.org/jira/browse/YARN-6299
> Project: Hadoop YARN
> Issue Type: Bug
> Components: fairscheduler
>Affects Versions: 3.0.0-alpha2
>Reporter: Daniel Templeton
>
> {code}
> Resource resourceUsage1 = s1.getResourceUsage();
> Resource resourceUsage2 = s2.getResourceUsage();
> Resource minShare1 = Resources.min(RESOURCE_CALCULATOR, null,
> s1.getMinShare(), s1.getDemand());
> Resource minShare2 = Resources.min(RESOURCE_CALCULATOR, null,
> s2.getMinShare(), s2.getDemand());
> boolean s1Needy = Resources.lessThan(RESOURCE_CALCULATOR, null,
> resourceUsage1, minShare1);
> boolean s2Needy = Resources.lessThan(RESOURCE_CALCULATOR, null,
> resourceUsage2, minShare2);
> minShareRatio1 = (double) resourceUsage1.getMemorySize()
> / Resources.max(RESOURCE_CALCULATOR, null, minShare1,
> ONE).getMemorySize();
> minShareRatio2 = (double) resourceUsage2.getMemorySize()
> / Resources.max(RESOURCE_CALCULATOR, null, minShare2,
> ONE).getMemorySize();
> {code}
> If demand is less than min share, then an app will be flagged as needy if it
> has demand that is higher than its usage, which happens any time the app has
> been assigned resources that it hasn't started using yet. That sounds wrong
> to me. [~kasha], [~yufeigu]?
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
