[jira] [Commented] (YARN-2380) The normalizeRequests method in SchedulerUtils always resets the vCore to 1
[
https://issues.apache.org/jira/browse/YARN-2380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14535547#comment-14535547
]
Zhijie Shen commented on YARN-2380:
---
I've two comments:
1. Other than normalize, DefaultResourceCalculator's other methods are still
using Resources.createResource(int), which will generate the resource object
with vcore = 0/1. This change makes the behavior inconsistent in
DefaultResourceCalculator.
2. There're failed test cases. And it bring me another question if users
provide vcore <= 0, do we need to reset them to 0/1 in this case?
> The normalizeRequests method in SchedulerUtils always resets the vCore to 1
> ---
>
> Key: YARN-2380
> URL: https://issues.apache.org/jira/browse/YARN-2380
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
>Affects Versions: 2.4.0
>Reporter: Jian Fang
>Assignee: Jian Fang
>Priority: Critical
> Attachments: YARN-2380.patch
>
>
> I added some log info to the method normalizeRequest() as follows.
> public static void normalizeRequest(
> ResourceRequest ask,
> ResourceCalculator resourceCalculator,
> Resource clusterResource,
> Resource minimumResource,
> Resource maximumResource,
> Resource incrementResource) {
> LOG.info("Before request normalization, the ask capacity: " +
> ask.getCapability());
> Resource normalized =
> Resources.normalize(
> resourceCalculator, ask.getCapability(), minimumResource,
> maximumResource, incrementResource);
> LOG.info("After request normalization, the ask capacity: " + normalized);
> ask.setCapability(normalized);
> }
> The resulted log showed that the vcore in ask was changed from 2 to 1.
> 2014-08-01 20:54:15,537 INFO
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils (IPC
> Server handler 4 on 9024): Before request normalization, the ask capacity:
>
> 2014-08-01 20:54:15,537 INFO
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils (IPC
> Server handler 4 on 9024): After request normalization, the ask capacity:
>
> The root cause is the DefaultResourceCalculator calls
> Resources.createResource(normalizedMemory) to regenerate a new resource with
> vcore = 1.
> This bug is critical and it leads to the mismatch of the request resource and
> the container resource and many other potential issues if the user requests
> containers with vcore > 1.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2380) The normalizeRequests method in SchedulerUtils always resets the vCore to 1
[
https://issues.apache.org/jira/browse/YARN-2380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14524110#comment-14524110
]
Hadoop QA commented on YARN-2380:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 14m 37s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:green}+1{color} | javac | 7m 30s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 37s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 22s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 1m 41s | There were no new checkstyle
issues. |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 32s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 32s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 2m 37s | The patch does not introduce
any new Findbugs (version 2.0.3) warnings. |
| {color:green}+1{color} | yarn tests | 1m 56s | Tests passed in
hadoop-yarn-common. |
| {color:red}-1{color} | yarn tests | 53m 47s | Tests failed in
hadoop-yarn-server-resourcemanager. |
| | | 94m 14s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests |
hadoop.yarn.server.resourcemanager.TestContainerResourceUsage |
| |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler |
| | hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesApps |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12661693/YARN-2380.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 3393461 |
| hadoop-yarn-common test log |
https://builds.apache.org/job/PreCommit-YARN-Build/7576/artifact/patchprocess/testrun_hadoop-yarn-common.txt
|
| hadoop-yarn-server-resourcemanager test log |
https://builds.apache.org/job/PreCommit-YARN-Build/7576/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/7576/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/7576/console |
This message was automatically generated.
> The normalizeRequests method in SchedulerUtils always resets the vCore to 1
> ---
>
> Key: YARN-2380
> URL: https://issues.apache.org/jira/browse/YARN-2380
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
>Affects Versions: 2.4.0
>Reporter: Jian Fang
>Assignee: Jian Fang
>Priority: Critical
> Attachments: YARN-2380.patch
>
>
> I added some log info to the method normalizeRequest() as follows.
> public static void normalizeRequest(
> ResourceRequest ask,
> ResourceCalculator resourceCalculator,
> Resource clusterResource,
> Resource minimumResource,
> Resource maximumResource,
> Resource incrementResource) {
> LOG.info("Before request normalization, the ask capacity: " +
> ask.getCapability());
> Resource normalized =
> Resources.normalize(
> resourceCalculator, ask.getCapability(), minimumResource,
> maximumResource, incrementResource);
> LOG.info("After request normalization, the ask capacity: " + normalized);
> ask.setCapability(normalized);
> }
> The resulted log showed that the vcore in ask was changed from 2 to 1.
> 2014-08-01 20:54:15,537 INFO
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils (IPC
> Server handler 4 on 9024): Before request normalization, the ask capacity:
>
> 2014-08-01 20:54:15,537 INFO
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils (IPC
> Server handler 4 on 9024): After request normalization, the ask capacity:
>
> The root cause is the DefaultResourceCalculator calls
> Resources.createResource(normalizedMemory) to regenerate a new resource with
> vcore = 1.
> This bug is critical and it leads to the mismatch of the request resource and
> the container resource and many other potential issues if the user requests
> containers with vcore > 1.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2380) The normalizeRequests method in SchedulerUtils always resets the vCore to 1
[
https://issues.apache.org/jira/browse/YARN-2380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14524104#comment-14524104
]
Hadoop QA commented on YARN-2380:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 14m 28s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:green}+1{color} | javac | 7m 28s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 36s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 22s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 1m 38s | There were no new checkstyle
issues. |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 34s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 32s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 2m 37s | The patch does not introduce
any new Findbugs (version 2.0.3) warnings. |
| {color:green}+1{color} | yarn tests | 1m 55s | Tests passed in
hadoop-yarn-common. |
| {color:red}-1{color} | yarn tests | 53m 42s | Tests failed in
hadoop-yarn-server-resourcemanager. |
| | | 93m 57s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler |
| | hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesApps |
| | hadoop.yarn.server.resourcemanager.TestContainerResourceUsage |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12661693/YARN-2380.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 3393461 |
| hadoop-yarn-common test log |
https://builds.apache.org/job/PreCommit-YARN-Build/7575/artifact/patchprocess/testrun_hadoop-yarn-common.txt
|
| hadoop-yarn-server-resourcemanager test log |
https://builds.apache.org/job/PreCommit-YARN-Build/7575/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/7575/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/7575/console |
This message was automatically generated.
> The normalizeRequests method in SchedulerUtils always resets the vCore to 1
> ---
>
> Key: YARN-2380
> URL: https://issues.apache.org/jira/browse/YARN-2380
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
>Affects Versions: 2.4.0
>Reporter: Jian Fang
>Assignee: Jian Fang
>Priority: Critical
> Attachments: YARN-2380.patch
>
>
> I added some log info to the method normalizeRequest() as follows.
> public static void normalizeRequest(
> ResourceRequest ask,
> ResourceCalculator resourceCalculator,
> Resource clusterResource,
> Resource minimumResource,
> Resource maximumResource,
> Resource incrementResource) {
> LOG.info("Before request normalization, the ask capacity: " +
> ask.getCapability());
> Resource normalized =
> Resources.normalize(
> resourceCalculator, ask.getCapability(), minimumResource,
> maximumResource, incrementResource);
> LOG.info("After request normalization, the ask capacity: " + normalized);
> ask.setCapability(normalized);
> }
> The resulted log showed that the vcore in ask was changed from 2 to 1.
> 2014-08-01 20:54:15,537 INFO
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils (IPC
> Server handler 4 on 9024): Before request normalization, the ask capacity:
>
> 2014-08-01 20:54:15,537 INFO
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils (IPC
> Server handler 4 on 9024): After request normalization, the ask capacity:
>
> The root cause is the DefaultResourceCalculator calls
> Resources.createResource(normalizedMemory) to regenerate a new resource with
> vcore = 1.
> This bug is critical and it leads to the mismatch of the request resource and
> the container resource and many other potential issues if the user requests
> containers with vcore > 1.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2380) The normalizeRequests method in SchedulerUtils always resets the vCore to 1
[
https://issues.apache.org/jira/browse/YARN-2380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14523877#comment-14523877
]
Junping Du commented on YARN-2380:
--
Not sure if test failure is related or not. Kick off Jenkins again manually.
> The normalizeRequests method in SchedulerUtils always resets the vCore to 1
> ---
>
> Key: YARN-2380
> URL: https://issues.apache.org/jira/browse/YARN-2380
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
>Affects Versions: 2.4.0
>Reporter: Jian Fang
>Assignee: Jian Fang
>Priority: Critical
> Attachments: YARN-2380.patch
>
>
> I added some log info to the method normalizeRequest() as follows.
> public static void normalizeRequest(
> ResourceRequest ask,
> ResourceCalculator resourceCalculator,
> Resource clusterResource,
> Resource minimumResource,
> Resource maximumResource,
> Resource incrementResource) {
> LOG.info("Before request normalization, the ask capacity: " +
> ask.getCapability());
> Resource normalized =
> Resources.normalize(
> resourceCalculator, ask.getCapability(), minimumResource,
> maximumResource, incrementResource);
> LOG.info("After request normalization, the ask capacity: " + normalized);
> ask.setCapability(normalized);
> }
> The resulted log showed that the vcore in ask was changed from 2 to 1.
> 2014-08-01 20:54:15,537 INFO
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils (IPC
> Server handler 4 on 9024): Before request normalization, the ask capacity:
>
> 2014-08-01 20:54:15,537 INFO
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils (IPC
> Server handler 4 on 9024): After request normalization, the ask capacity:
>
> The root cause is the DefaultResourceCalculator calls
> Resources.createResource(normalizedMemory) to regenerate a new resource with
> vcore = 1.
> This bug is critical and it leads to the mismatch of the request resource and
> the container resource and many other potential issues if the user requests
> containers with vcore > 1.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2380) The normalizeRequests method in SchedulerUtils always resets the vCore to 1
[
https://issues.apache.org/jira/browse/YARN-2380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14345776#comment-14345776
]
Wangda Tan commented on YARN-2380:
--
[~kj-ki],
Thought about it, similar problems existed in other methods like
roundUp/divideAndCeil, etc. probably we should make their behavior consistent.
My proposal is, if we keep vcore when normalize, we need do math on vcore when
call roundUp/divideAndCeil, etc. We will only ignore vcore when doing compare
operations.
Make sense?
> The normalizeRequests method in SchedulerUtils always resets the vCore to 1
> ---
>
> Key: YARN-2380
> URL: https://issues.apache.org/jira/browse/YARN-2380
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
>Affects Versions: 2.4.0
>Reporter: Jian Fang
>Priority: Critical
> Attachments: YARN-2380.patch
>
>
> I added some log info to the method normalizeRequest() as follows.
> public static void normalizeRequest(
> ResourceRequest ask,
> ResourceCalculator resourceCalculator,
> Resource clusterResource,
> Resource minimumResource,
> Resource maximumResource,
> Resource incrementResource) {
> LOG.info("Before request normalization, the ask capacity: " +
> ask.getCapability());
> Resource normalized =
> Resources.normalize(
> resourceCalculator, ask.getCapability(), minimumResource,
> maximumResource, incrementResource);
> LOG.info("After request normalization, the ask capacity: " + normalized);
> ask.setCapability(normalized);
> }
> The resulted log showed that the vcore in ask was changed from 2 to 1.
> 2014-08-01 20:54:15,537 INFO
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils (IPC
> Server handler 4 on 9024): Before request normalization, the ask capacity:
>
> 2014-08-01 20:54:15,537 INFO
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils (IPC
> Server handler 4 on 9024): After request normalization, the ask capacity:
>
> The root cause is the DefaultResourceCalculator calls
> Resources.createResource(normalizedMemory) to regenerate a new resource with
> vcore = 1.
> This bug is critical and it leads to the mismatch of the request resource and
> the container resource and many other potential issues if the user requests
> containers with vcore > 1.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2380) The normalizeRequests method in SchedulerUtils always resets the vCore to 1
[
https://issues.apache.org/jira/browse/YARN-2380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1439#comment-1439
]
Hadoop QA commented on YARN-2380:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12661693/YARN-2380.patch
against trunk revision 431e7d8.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 5 new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesApps
org.apache.hadoop.yarn.server.resourcemanager.TestContainerResourceUsage
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/6812//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-YARN-Build/6812//artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6812//console
This message is automatically generated.
> The normalizeRequests method in SchedulerUtils always resets the vCore to 1
> ---
>
> Key: YARN-2380
> URL: https://issues.apache.org/jira/browse/YARN-2380
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
>Affects Versions: 2.4.0
>Reporter: Jian Fang
>Priority: Critical
> Attachments: YARN-2380.patch
>
>
> I added some log info to the method normalizeRequest() as follows.
> public static void normalizeRequest(
> ResourceRequest ask,
> ResourceCalculator resourceCalculator,
> Resource clusterResource,
> Resource minimumResource,
> Resource maximumResource,
> Resource incrementResource) {
> LOG.info("Before request normalization, the ask capacity: " +
> ask.getCapability());
> Resource normalized =
> Resources.normalize(
> resourceCalculator, ask.getCapability(), minimumResource,
> maximumResource, incrementResource);
> LOG.info("After request normalization, the ask capacity: " + normalized);
> ask.setCapability(normalized);
> }
> The resulted log showed that the vcore in ask was changed from 2 to 1.
> 2014-08-01 20:54:15,537 INFO
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils (IPC
> Server handler 4 on 9024): Before request normalization, the ask capacity:
>
> 2014-08-01 20:54:15,537 INFO
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils (IPC
> Server handler 4 on 9024): After request normalization, the ask capacity:
>
> The root cause is the DefaultResourceCalculator calls
> Resources.createResource(normalizedMemory) to regenerate a new resource with
> vcore = 1.
> This bug is critical and it leads to the mismatch of the request resource and
> the container resource and many other potential issues if the user requests
> containers with vcore > 1.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2380) The normalizeRequests method in SchedulerUtils always resets the vCore to 1
[
https://issues.apache.org/jira/browse/YARN-2380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14344261#comment-14344261
]
Wangda Tan commented on YARN-2380:
--
Patch looks good to me, will commit it when Jenkins get back.
> The normalizeRequests method in SchedulerUtils always resets the vCore to 1
> ---
>
> Key: YARN-2380
> URL: https://issues.apache.org/jira/browse/YARN-2380
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
>Affects Versions: 2.4.0
>Reporter: Jian Fang
>Priority: Critical
> Attachments: YARN-2380.patch
>
>
> I added some log info to the method normalizeRequest() as follows.
> public static void normalizeRequest(
> ResourceRequest ask,
> ResourceCalculator resourceCalculator,
> Resource clusterResource,
> Resource minimumResource,
> Resource maximumResource,
> Resource incrementResource) {
> LOG.info("Before request normalization, the ask capacity: " +
> ask.getCapability());
> Resource normalized =
> Resources.normalize(
> resourceCalculator, ask.getCapability(), minimumResource,
> maximumResource, incrementResource);
> LOG.info("After request normalization, the ask capacity: " + normalized);
> ask.setCapability(normalized);
> }
> The resulted log showed that the vcore in ask was changed from 2 to 1.
> 2014-08-01 20:54:15,537 INFO
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils (IPC
> Server handler 4 on 9024): Before request normalization, the ask capacity:
>
> 2014-08-01 20:54:15,537 INFO
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils (IPC
> Server handler 4 on 9024): After request normalization, the ask capacity:
>
> The root cause is the DefaultResourceCalculator calls
> Resources.createResource(normalizedMemory) to regenerate a new resource with
> vcore = 1.
> This bug is critical and it leads to the mismatch of the request resource and
> the container resource and many other potential issues if the user requests
> containers with vcore > 1.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2380) The normalizeRequests method in SchedulerUtils always resets the vCore to 1
[
https://issues.apache.org/jira/browse/YARN-2380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14097225#comment-14097225
]
Jian Fang commented on YARN-2380:
-
Any reason why you think DefaultResourceCalculator should care only about
memory?
Up to now, the resource is defined as . If
DefaultResourceCalculator does not do anything about memory, it should pass
through the vcore value instead of setting it to 1, which would lead to a lot
of potential issues such as the case in Tez.
> The normalizeRequests method in SchedulerUtils always resets the vCore to 1
> ---
>
> Key: YARN-2380
> URL: https://issues.apache.org/jira/browse/YARN-2380
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
>Affects Versions: 2.4.0
>Reporter: Jian Fang
>Priority: Critical
> Attachments: YARN-2380.patch
>
>
> I added some log info to the method normalizeRequest() as follows.
> public static void normalizeRequest(
> ResourceRequest ask,
> ResourceCalculator resourceCalculator,
> Resource clusterResource,
> Resource minimumResource,
> Resource maximumResource,
> Resource incrementResource) {
> LOG.info("Before request normalization, the ask capacity: " +
> ask.getCapability());
> Resource normalized =
> Resources.normalize(
> resourceCalculator, ask.getCapability(), minimumResource,
> maximumResource, incrementResource);
> LOG.info("After request normalization, the ask capacity: " + normalized);
> ask.setCapability(normalized);
> }
> The resulted log showed that the vcore in ask was changed from 2 to 1.
> 2014-08-01 20:54:15,537 INFO
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils (IPC
> Server handler 4 on 9024): Before request normalization, the ask capacity:
>
> 2014-08-01 20:54:15,537 INFO
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils (IPC
> Server handler 4 on 9024): After request normalization, the ask capacity:
>
> The root cause is the DefaultResourceCalculator calls
> Resources.createResource(normalizedMemory) to regenerate a new resource with
> vcore = 1.
> This bug is critical and it leads to the mismatch of the request resource and
> the container resource and many other potential issues if the user requests
> containers with vcore > 1.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2380) The normalizeRequests method in SchedulerUtils always resets the vCore to 1
[
https://issues.apache.org/jira/browse/YARN-2380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14083171#comment-14083171
]
Wei Yan commented on YARN-2380:
---
[~john.jian.fang], maybe you can try FairScheduler which uses
DominantResourceCalculator for normalization, before the problem fixed.
> The normalizeRequests method in SchedulerUtils always resets the vCore to 1
> ---
>
> Key: YARN-2380
> URL: https://issues.apache.org/jira/browse/YARN-2380
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
>Affects Versions: 2.4.0
>Reporter: Jian Fang
>Priority: Critical
>
> I added some log info to the method normalizeRequest() as follows.
> public static void normalizeRequest(
> ResourceRequest ask,
> ResourceCalculator resourceCalculator,
> Resource clusterResource,
> Resource minimumResource,
> Resource maximumResource,
> Resource incrementResource) {
> LOG.info("Before request normalization, the ask capacity: " +
> ask.getCapability());
> Resource normalized =
> Resources.normalize(
> resourceCalculator, ask.getCapability(), minimumResource,
> maximumResource, incrementResource);
> LOG.info("After request normalization, the ask capacity: " + normalized);
> ask.setCapability(normalized);
> }
> The resulted log showed that the vcore in ask was changed from 2 to 1.
> 2014-08-01 20:54:15,537 INFO
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils (IPC
> Server handler 4 on 9024): Before request normalization, the ask capacity:
>
> 2014-08-01 20:54:15,537 INFO
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils (IPC
> Server handler 4 on 9024): After request normalization, the ask capacity:
>
> The root cause is the DefaultResourceCalculator calls
> Resources.createResource(normalizedMemory) to regenerate a new resource with
> vcore = 1.
> This bug is critical and it leads to the mismatch of the request resource and
> the container resource and many other potential issues if the user requests
> containers with vcore > 1.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2380) The normalizeRequests method in SchedulerUtils always resets the vCore to 1
[
https://issues.apache.org/jira/browse/YARN-2380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14083164#comment-14083164
]
Jian Fang commented on YARN-2380:
-
Seems DominantResourceCalculator sets vcore correctly. But the capacity
scheduler in YARN uses DefaultResourceCalculator for
yarn.scheduler.capacity.resource-calculator by default.
Anyway, resetting vcore to 1 is an unexpected behavior, which could create a
lot of troubles. Either fix DefaultResourceCalculator or just remove it.
> The normalizeRequests method in SchedulerUtils always resets the vCore to 1
> ---
>
> Key: YARN-2380
> URL: https://issues.apache.org/jira/browse/YARN-2380
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
>Affects Versions: 2.4.0
>Reporter: Jian Fang
>Priority: Critical
>
> I added some log info to the method normalizeRequest() as follows.
> public static void normalizeRequest(
> ResourceRequest ask,
> ResourceCalculator resourceCalculator,
> Resource clusterResource,
> Resource minimumResource,
> Resource maximumResource,
> Resource incrementResource) {
> LOG.info("Before request normalization, the ask capacity: " +
> ask.getCapability());
> Resource normalized =
> Resources.normalize(
> resourceCalculator, ask.getCapability(), minimumResource,
> maximumResource, incrementResource);
> LOG.info("After request normalization, the ask capacity: " + normalized);
> ask.setCapability(normalized);
> }
> The resulted log showed that the vcore in ask was changed from 2 to 1.
> 2014-08-01 20:54:15,537 INFO
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils (IPC
> Server handler 4 on 9024): Before request normalization, the ask capacity:
>
> 2014-08-01 20:54:15,537 INFO
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils (IPC
> Server handler 4 on 9024): After request normalization, the ask capacity:
>
> The root cause is the DefaultResourceCalculator calls
> Resources.createResource(normalizedMemory) to regenerate a new resource with
> vcore = 1.
> This bug is critical and it leads to the mismatch of the request resource and
> the container resource and many other potential issues if the user requests
> containers with vcore > 1.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
