[jira] [Comment Edited] (YARN-6428) Queue AM limit is not honored in CS always
[
https://issues.apache.org/jira/browse/YARN-6428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16036779#comment-16036779
]
Bibin A Chundatt edited comment on YARN-6428 at 6/5/17 10:28 AM:
-
During offline discussion with [~naganarasimha...@apache.org] for
{code}
@Override
public Resource multiplyAndNormalizeUp(Resource r, double by,
Resource stepFactor) {
return Resources.createResource(
roundUp(
(int)Math.ceil(r.getMemorySize() * by), stepFactor.getMemorySize()),
roundUp(
(int)Math.ceil(r.getVirtualCores() * by),
stepFactor.getVirtualCores())
);
}
{code}
In the above base code instead off {{(int)Math.ceil(r.getMemorySize() * by)}}
it should be {{(long)Math.ceil(r.getMemorySize() * by)}} . For
*DefaultResourceCalculator* {{long}} is used but *DominantResourceCalculator*
{{int}} is used.Seems to be a miss during type datatype change
was (Author: bibinchundatt):
Adding point we had in offline discussion with [~naganarasimha...@apache.org]
{code}
@Override
public Resource multiplyAndNormalizeUp(Resource r, double by,
Resource stepFactor) {
return Resources.createResource(
roundUp(
(int)Math.ceil(r.getMemorySize() * by), stepFactor.getMemorySize()),
roundUp(
(int)Math.ceil(r.getVirtualCores() * by),
stepFactor.getVirtualCores())
);
}
{code}
In the above base code instead off {{(int)Math.ceil(r.getMemorySize() * by)}}
it should be {{(long)Math.ceil(r.getMemorySize() * by)}} . For DefaultResource
Calculator {{long}} is used but *DefaultResourceCalculator* {{int}} is used.
> Queue AM limit is not honored in CS always
> ---
>
> Key: YARN-6428
> URL: https://issues.apache.org/jira/browse/YARN-6428
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
> Attachments: YARN-6428.0001.patch
>
>
> Steps to reproduce
>
> Setup cluster with 40 GB and 40 vcores with 4 Node managers with 10 GB each.
> Configure 100% to default queue as capacity and max am limit as 10 %
> Minimum scheduler memory and vcore as 512,1
> *Expected*
> AM limit 4096 and 4 vores
> *Actual*
> AM limit 4096+512 and 4+1 vcore
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-6428) Queue AM limit is not honored in CS always
[ https://issues.apache.org/jira/browse/YARN-6428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15978691#comment-15978691 ] Bibin A Chundatt edited comment on YARN-6428 at 4/25/17 4:43 PM: - [~sunilg] / [~leftnoteasy] Could you please share your thoughts.? was (Author: bibinchundatt): [~sunilg] / [~leftnoteasy] Any comments? > Queue AM limit is not honored in CS always > --- > > Key: YARN-6428 > URL: https://issues.apache.org/jira/browse/YARN-6428 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Bibin A Chundatt >Assignee: Bibin A Chundatt > Attachments: YARN-6428.0001.patch > > > Steps to reproduce > > Setup cluster with 40 GB and 40 vcores with 4 Node managers with 10 GB each. > Configure 100% to default queue as capacity and max am limit as 10 % > Minimum scheduler memory and vcore as 512,1 > *Expected* > AM limit 4096 and 4 vores > *Actual* > AM limit 4096+512 and 4+1 vcore -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-6428) Queue AM limit is not honored in CS always
[
https://issues.apache.org/jira/browse/YARN-6428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15953940#comment-15953940
]
Bibin A Chundatt edited comment on YARN-6428 at 4/3/17 6:18 PM:
{{DefaultResourceCalculator}} any idea why {{0.5}} is added don't find the same
in {{DominantResourceCalculator}}
{code}
@Override
public Resource multiplyAndNormalizeUp(Resource r, double by,
Resource stepFactor) {
return Resources.createResource(
roundUp((long) (r.getMemorySize() * by + 0.5),
stepFactor.getMemorySize()));
}
{code}
We should use {{Math.ceil}} here too.
was (Author: bibinchundatt):
{{DefaultResourceCalculator}} any idea why {{0.5}} is added don't find the same
in {{DominantResourceCalculator}}
{code}
@Override
public Resource multiplyAndNormalizeUp(Resource r, double by,
Resource stepFactor) {
return Resources.createResource(
roundUp((long) (r.getMemorySize() * by + 0.5),
stepFactor.getMemorySize()));
}
{code}
> Queue AM limit is not honored in CS always
> ---
>
> Key: YARN-6428
> URL: https://issues.apache.org/jira/browse/YARN-6428
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
> Attachments: YARN-6428.0001.patch
>
>
> Steps to reproduce
>
> Setup cluster with 40 GB and 40 vcores with 4 Node managers with 10 GB each.
> Configure 100% to default queue as capacity and max am limit as 10 %
> Minimum scheduler memory and vcore as 512,1
> *Expected*
> AM limit 4096 and 4 vores
> *Actual*
> AM limit 4096+512 and 4+1 vcore
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-6428) Queue AM limit is not honored in CS always
[
https://issues.apache.org/jira/browse/YARN-6428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15953068#comment-15953068
]
Bibin A Chundatt edited comment on YARN-6428 at 4/3/17 12:44 PM:
-
{{ResourceCalculator#multiplyAndNormalizeUp}} using double to multiply and ceil
for am limit factor.
{code}
roundUp(
(int)Math.ceil(r.getMemorySize() * by)
{code}
{{0.1}} will be {{0.1012312444}} so multiple and ceil and round by will
give give 4097 instead of 4096.
was (Author: bibinchundatt):
{{ResourceCalculator#multiplyAndNormalizeUp}} using double to multiply and ceil
for am limit factor.
{code}
roundUp(
(int)Math.ceil(r.getMemorySize() * by)
{code}
will give 4097 instead of 4096 .
> Queue AM limit is not honored in CS always
> ---
>
> Key: YARN-6428
> URL: https://issues.apache.org/jira/browse/YARN-6428
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
> Attachments: YARN-6428.0001.patch
>
>
> Steps to reproduce
>
> Setup cluster with 40 GB and 40 vcores with 4 Node managers with 10 GB each.
> Configure 100% to default queue as capacity and max am limit as 10 %
> Minimum scheduler memory and vcore as 512,1
> *Expected*
> AM limit 4096 and 4 vores
> *Actual*
> AM limit 4096+512 and 4+1 vcore
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-6428) Queue AM limit is not honored in CS always
[
https://issues.apache.org/jira/browse/YARN-6428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15953405#comment-15953405
]
Bibin A Chundatt edited comment on YARN-6428 at 4/3/17 12:43 PM:
-
[~varun_saxena]
Thank you for looking into the issue. The second approach will be a bigger
change than
current. for example: {{QueueCapacities}} currently uses float .
In current implementation we have option to used both interface.
was (Author: bibinchundatt):
[~varun_saxena]
Thank you for looking into the issue. The second approach will be a bigger
change that current. for example: {{QueueCapacities}} currently uses float .
In current implementation we have option to used both interface.
> Queue AM limit is not honored in CS always
> ---
>
> Key: YARN-6428
> URL: https://issues.apache.org/jira/browse/YARN-6428
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
> Attachments: YARN-6428.0001.patch
>
>
> Steps to reproduce
>
> Setup cluster with 40 GB and 40 vcores with 4 Node managers with 10 GB each.
> Configure 100% to default queue as capacity and max am limit as 10 %
> Minimum scheduler memory and vcore as 512,1
> *Expected*
> AM limit 4096 and 4 vores
> *Actual*
> AM limit 4096+512 and 4+1 vcore
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
