[jira] [Commented] (YARN-5795) FairScheduler set AppMaster vcores didn't work
[
https://issues.apache.org/jira/browse/YARN-5795?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15691743#comment-15691743
]
Yufei Gu commented on YARN-5795:
Yes. I found this issue as well. No matter how many vcores you give it to
resource request, {{DefaultResourceCalculator}} will assign 1 to it while
normalizing.
It is hard to say if this is bug for {{DefaultResourceCalculator}} since it
depends on how/when we use it.
This is more about an issue of fair scheduler. In FairScheduler, we use the
{{DominantResourceCalculator}} in the most case, but the weird thing is if your
invoke {{fairScheduler.getResourceCalculator()}}, you will get a
{{DefaultResourceCalculator}}. That doesn't sound reasonable to me. We should
investigate and make some changes.
YARN-5774 can solve this issue incidentally. Maybe we could focus on the what
should {{fairScheduler.getResourceCalculator()}} return in this JIRA.
> FairScheduler set AppMaster vcores didn't work
> --
>
> Key: YARN-5795
> URL: https://issues.apache.org/jira/browse/YARN-5795
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
>Affects Versions: 2.7.1
>Reporter: tangshangwen
>Assignee: tangshangwen
> Attachments: 0001-YARN-5795.patch
>
>
> In our cluster, we use the linux container, I would like to increase the
> number of cpu to get more CPU time slice, but it did not take effect， i set
> yarn.app.mapreduce.am.resource.cpu-vcores = 2 ,but i found the
> resourcemanager log
> {noformat}
> [2016-10-27T16:36:37.280 08:00] [INFO]
> resourcemanager.scheduler.SchedulerNode.allocateContainer(SchedulerNode.java
> 153) [ResourceManager Event Processor] : Assigned container
> container_1477059529836_336635_01_01 of capacity
> {noformat}
> Because scheduler.getResourceCalculator() only computes memory
> {code:title=RMAppManager.java|borderStyle=solid}
> private ResourceRequest validateAndCreateResourceRequest(
> ApplicationSubmissionContext submissionContext, boolean isRecovery)
> throws InvalidResourceRequestException {
> ...
> SchedulerUtils.normalizeRequest(amReq,
> scheduler.getResourceCalculator(),
> scheduler.getClusterResource(),
> scheduler.getMinimumResourceCapability(),
> scheduler.getMaximumResourceCapability(),
> scheduler.getMinimumResourceCapability());
> ...
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5795) FairScheduler set AppMaster vcores didn't work
[
https://issues.apache.org/jira/browse/YARN-5795?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15690456#comment-15690456
]
Daniel Templeton commented on YARN-5795:
[~yufeigu] is in the process of overhauling the resource normalization process
in YARN-5774. At a minimum, we should wait until that change is in. It would
be nice, though, to also fix this issue in that patch.
> FairScheduler set AppMaster vcores didn't work
> --
>
> Key: YARN-5795
> URL: https://issues.apache.org/jira/browse/YARN-5795
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
>Affects Versions: 2.7.1
>Reporter: tangshangwen
>Assignee: tangshangwen
> Attachments: 0001-YARN-5795.patch
>
>
> In our cluster, we use the linux container, I would like to increase the
> number of cpu to get more CPU time slice, but it did not take effect， i set
> yarn.app.mapreduce.am.resource.cpu-vcores = 2 ,but i found the
> resourcemanager log
> {noformat}
> [2016-10-27T16:36:37.280 08:00] [INFO]
> resourcemanager.scheduler.SchedulerNode.allocateContainer(SchedulerNode.java
> 153) [ResourceManager Event Processor] : Assigned container
> container_1477059529836_336635_01_01 of capacity
> {noformat}
> Because scheduler.getResourceCalculator() only computes memory
> {code:title=RMAppManager.java|borderStyle=solid}
> private ResourceRequest validateAndCreateResourceRequest(
> ApplicationSubmissionContext submissionContext, boolean isRecovery)
> throws InvalidResourceRequestException {
> ...
> SchedulerUtils.normalizeRequest(amReq,
> scheduler.getResourceCalculator(),
> scheduler.getClusterResource(),
> scheduler.getMinimumResourceCapability(),
> scheduler.getMaximumResourceCapability(),
> scheduler.getMinimumResourceCapability());
> ...
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5795) FairScheduler set AppMaster vcores didn't work
[
https://issues.apache.org/jira/browse/YARN-5795?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15657056#comment-15657056
]
tangshangwen commented on YARN-5795:
Hi [~templedf], would you like to review the patch or give me some pointer for
the next step to do ?
> FairScheduler set AppMaster vcores didn't work
> --
>
> Key: YARN-5795
> URL: https://issues.apache.org/jira/browse/YARN-5795
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
>Affects Versions: 2.7.1
>Reporter: tangshangwen
>Assignee: tangshangwen
> Attachments: 0001-YARN-5795.patch
>
>
> In our cluster, we use the linux container, I would like to increase the
> number of cpu to get more CPU time slice, but it did not take effect， i set
> yarn.app.mapreduce.am.resource.cpu-vcores = 2 ,but i found the
> resourcemanager log
> {noformat}
> [2016-10-27T16:36:37.280 08:00] [INFO]
> resourcemanager.scheduler.SchedulerNode.allocateContainer(SchedulerNode.java
> 153) [ResourceManager Event Processor] : Assigned container
> container_1477059529836_336635_01_01 of capacity
> {noformat}
> Because scheduler.getResourceCalculator() only computes memory
> {code:title=RMAppManager.java|borderStyle=solid}
> private ResourceRequest validateAndCreateResourceRequest(
> ApplicationSubmissionContext submissionContext, boolean isRecovery)
> throws InvalidResourceRequestException {
> ...
> SchedulerUtils.normalizeRequest(amReq,
> scheduler.getResourceCalculator(),
> scheduler.getClusterResource(),
> scheduler.getMinimumResourceCapability(),
> scheduler.getMaximumResourceCapability(),
> scheduler.getMinimumResourceCapability());
> ...
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5795) FairScheduler set AppMaster vcores didn't work
[
https://issues.apache.org/jira/browse/YARN-5795?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15614804#comment-15614804
]
tangshangwen commented on YARN-5795:
hi [~kasha], i think the DefaultResourceCalculator in the normalize method
should be to maintain the original vcore,
{code:title=DefaultResourceCalculator.java|borderStyle=solid}
@Override
public Resource normalize(Resource r, Resource minimumResource,
Resource maximumResource, Resource stepFactor) {
int normalizedMemory = Math.min(
roundUp(
Math.max(r.getMemory(), minimumResource.getMemory()),
stepFactor.getMemory()),
maximumResource.getMemory());
return Resources.createResource(normalizedMemory);
}
{code}
change to
{code:title=DefaultResourceCalculator.java|borderStyle=solid}
@Override
public Resource normalize(Resource r, Resource minimumResource,
Resource maximumResource, Resource stepFactor) {
int normalizedMemory = Math.min(
roundUp(
Math.max(r.getMemory(), minimumResource.getMemory()),
stepFactor.getMemory()),
maximumResource.getMemory());
return Resources.createResource(normalizedMemory, r.getVirtualCores());
}
{code}
> FairScheduler set AppMaster vcores didn't work
> --
>
> Key: YARN-5795
> URL: https://issues.apache.org/jira/browse/YARN-5795
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
>Affects Versions: 2.7.1
>Reporter: tangshangwen
>Assignee: tangshangwen
>
> In our cluster, we use the linux container, I would like to increase the
> number of cpu to get more CPU time slice, but it did not take effect， i set
> yarn.app.mapreduce.am.resource.cpu-vcores = 2 ,but i found the
> resourcemanager log
> {noformat}
> [2016-10-27T16:36:37.280 08:00] [INFO]
> resourcemanager.scheduler.SchedulerNode.allocateContainer(SchedulerNode.java
> 153) [ResourceManager Event Processor] : Assigned container
> container_1477059529836_336635_01_01 of capacity
> {noformat}
> Because scheduler.getResourceCalculator() only computes memory
> {code:title=RMAppManager.java|borderStyle=solid}
> private ResourceRequest validateAndCreateResourceRequest(
> ApplicationSubmissionContext submissionContext, boolean isRecovery)
> throws InvalidResourceRequestException {
> ...
> SchedulerUtils.normalizeRequest(amReq,
> scheduler.getResourceCalculator(),
> scheduler.getClusterResource(),
> scheduler.getMinimumResourceCapability(),
> scheduler.getMaximumResourceCapability(),
> scheduler.getMinimumResourceCapability());
> ...
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5795) FairScheduler set AppMaster vcores didn't work
[
https://issues.apache.org/jira/browse/YARN-5795?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15614196#comment-15614196
]
tangshangwen commented on YARN-5795:
I think allocate method using DOMINANT RESOURCE CALCULATOR, AppMaster should
also be the same.
{code:title=FairScheduler.java|borderStyle=solid}
@Override
public Allocation allocate(ApplicationAttemptId appAttemptId,
List ask, List release,
List blacklistAdditions, List blacklistRemovals) {
// Make sure this application exists
FSAppAttempt application = getSchedulerApp(appAttemptId);
if (application == null) {
LOG.info("Calling allocate on removed " +
"or non existant application " + appAttemptId);
return EMPTY_ALLOCATION;
}
// Sanity check
SchedulerUtils.normalizeRequests(ask, DOMINANT_RESOURCE_CALCULATOR,
clusterResource, minimumAllocation, getMaximumResourceCapability(),
incrAllocation);
..
}
{code}
> FairScheduler set AppMaster vcores didn't work
> --
>
> Key: YARN-5795
> URL: https://issues.apache.org/jira/browse/YARN-5795
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
>Affects Versions: 2.7.1
>Reporter: tangshangwen
>Assignee: tangshangwen
>
> In our cluster, we use the linux container, I would like to increase the
> number of cpu to get more CPU time slice, but it did not take effect， i set
> yarn.app.mapreduce.am.resource.cpu-vcores = 2 ,but i found the
> resourcemanager log
> {noformat}
> [2016-10-27T16:36:37.280 08:00] [INFO]
> resourcemanager.scheduler.SchedulerNode.allocateContainer(SchedulerNode.java
> 153) [ResourceManager Event Processor] : Assigned container
> container_1477059529836_336635_01_01 of capacity
> {noformat}
> Because scheduler.getResourceCalculator() only computes memory
> {code:title=RMAppManager.java|borderStyle=solid}
> private ResourceRequest validateAndCreateResourceRequest(
> ApplicationSubmissionContext submissionContext, boolean isRecovery)
> throws InvalidResourceRequestException {
> ...
> SchedulerUtils.normalizeRequest(amReq,
> scheduler.getResourceCalculator(),
> scheduler.getClusterResource(),
> scheduler.getMinimumResourceCapability(),
> scheduler.getMaximumResourceCapability(),
> scheduler.getMinimumResourceCapability());
> ...
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5795) FairScheduler set AppMaster vcores didn't work
[
https://issues.apache.org/jira/browse/YARN-5795?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15614149#comment-15614149
]
tangshangwen commented on YARN-5795:
I think if we replace RESOURCE_CALCULATOR with DOMINANT_RESOURCE_CALCULATOR we
should be able to fix the problem
{code:title=FairScheduler.java|borderStyle=solid}
@Override
public ResourceCalculator getResourceCalculator() {
//return RESOURCE_CALCULATOR;
return DOMINANT_RESOURCE_CALCULATOR;
}
{code}
> FairScheduler set AppMaster vcores didn't work
> --
>
> Key: YARN-5795
> URL: https://issues.apache.org/jira/browse/YARN-5795
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
>Affects Versions: 2.7.1
>Reporter: tangshangwen
>Assignee: tangshangwen
>
> In our cluster, we use the linux container, I would like to increase the
> number of cpu to get more CPU time slice, but it did not take effect， i set
> yarn.app.mapreduce.am.resource.cpu-vcores = 2 ,but i found the
> resourcemanager log
> {noformat}
> [2016-10-27T16:36:37.280 08:00] [INFO]
> resourcemanager.scheduler.SchedulerNode.allocateContainer(SchedulerNode.java
> 153) [ResourceManager Event Processor] : Assigned container
> container_1477059529836_336635_01_01 of capacity
> {noformat}
> Because scheduler.getResourceCalculator() only computes memory
> {code:title=RMAppManager.java|borderStyle=solid}
> private ResourceRequest validateAndCreateResourceRequest(
> ApplicationSubmissionContext submissionContext, boolean isRecovery)
> throws InvalidResourceRequestException {
> ...
> SchedulerUtils.normalizeRequest(amReq,
> scheduler.getResourceCalculator(),
> scheduler.getClusterResource(),
> scheduler.getMinimumResourceCapability(),
> scheduler.getMaximumResourceCapability(),
> scheduler.getMinimumResourceCapability());
> ...
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
