[jira] [Commented] (YARN-3733) DominantRC#compare() does not work as expected if cluster resource is empty
[ https://issues.apache.org/jira/browse/YARN-3733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14572239#comment-14572239 ] Sunil G commented on YARN-3733: --- Patch looks good to me. +1 For MockRM.submitApp, I think we need to support the addition of Cores and Memory. I will file a separate ticket to handle the same if its fine. DominantRC#compare() does not work as expected if cluster resource is empty --- Key: YARN-3733 URL: https://issues.apache.org/jira/browse/YARN-3733 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager Affects Versions: 2.7.0 Environment: Suse 11 Sp3 , 2 NM , 2 RM one NM - 3 GB 6 v core Reporter: Bibin A Chundatt Assignee: Rohith Priority: Blocker Attachments: 0001-YARN-3733.patch, 0002-YARN-3733.patch, 0002-YARN-3733.patch, YARN-3733.patch Steps to reproduce = 1. Install HA with 2 RM 2 NM (3072 MB * 2 total cluster) 2. Configure map and reduce size to 512 MB after changing scheduler minimum size to 512 MB 3. Configure capacity scheduler and AM limit to .5 (DominantResourceCalculator is configured) 4. Submit 30 concurrent task 5. Switch RM Actual = For 12 Jobs AM gets allocated and all 12 starts running No other Yarn child is initiated , *all 12 Jobs in Running state for ever* Expected === Only 6 should be running at a time since max AM allocated is .5 (3072 MB) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3733) DominantRC#compare() does not work as expected if cluster resource is empty
[ https://issues.apache.org/jira/browse/YARN-3733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14572247#comment-14572247 ] Rohith commented on YARN-3733: -- +1 for handling virtual core's. This will good immprovement for testing DominantRC functionality precicely. DominantRC#compare() does not work as expected if cluster resource is empty --- Key: YARN-3733 URL: https://issues.apache.org/jira/browse/YARN-3733 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager Affects Versions: 2.7.0 Environment: Suse 11 Sp3 , 2 NM , 2 RM one NM - 3 GB 6 v core Reporter: Bibin A Chundatt Assignee: Rohith Priority: Blocker Attachments: 0001-YARN-3733.patch, 0002-YARN-3733.patch, 0002-YARN-3733.patch, YARN-3733.patch Steps to reproduce = 1. Install HA with 2 RM 2 NM (3072 MB * 2 total cluster) 2. Configure map and reduce size to 512 MB after changing scheduler minimum size to 512 MB 3. Configure capacity scheduler and AM limit to .5 (DominantResourceCalculator is configured) 4. Submit 30 concurrent task 5. Switch RM Actual = For 12 Jobs AM gets allocated and all 12 starts running No other Yarn child is initiated , *all 12 Jobs in Running state for ever* Expected === Only 6 should be running at a time since max AM allocated is .5 (3072 MB) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3733) DominantRC#compare() does not work as expected if cluster resource is empty
[
https://issues.apache.org/jira/browse/YARN-3733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=1457#comment-1457
]
Hadoop QA commented on YARN-3733:
-
\\
\\
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 17m 28s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 2 new or modified test files. |
| {color:green}+1{color} | javac | 7m 36s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 39s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 23s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 1m 40s | There were no new checkstyle
issues. |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 36s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 34s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 2m 59s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | yarn tests | 1m 56s | Tests passed in
hadoop-yarn-common. |
| {color:green}+1{color} | yarn tests | 50m 24s | Tests passed in
hadoop-yarn-server-resourcemanager. |
| | | 94m 18s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12737453/0002-YARN-3733.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / b5f0d29 |
| hadoop-yarn-common test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8191/artifact/patchprocess/testrun_hadoop-yarn-common.txt
|
| hadoop-yarn-server-resourcemanager test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8191/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/8191/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf907.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/8191/console |
This message was automatically generated.
DominantRC#compare() does not work as expected if cluster resource is empty
---
Key: YARN-3733
URL: https://issues.apache.org/jira/browse/YARN-3733
Project: Hadoop YARN
Issue Type: Bug
Components: resourcemanager
Affects Versions: 2.7.0
Environment: Suse 11 Sp3 , 2 NM , 2 RM
one NM - 3 GB 6 v core
Reporter: Bibin A Chundatt
Assignee: Rohith
Priority: Blocker
Attachments: 0001-YARN-3733.patch, 0002-YARN-3733.patch,
0002-YARN-3733.patch, YARN-3733.patch
Steps to reproduce
=
1. Install HA with 2 RM 2 NM (3072 MB * 2 total cluster)
2. Configure map and reduce size to 512 MB after changing scheduler minimum
size to 512 MB
3. Configure capacity scheduler and AM limit to .5
(DominantResourceCalculator is configured)
4. Submit 30 concurrent task
5. Switch RM
Actual
=
For 12 Jobs AM gets allocated and all 12 starts running
No other Yarn child is initiated , *all 12 Jobs in Running state for ever*
Expected
===
Only 6 should be running at a time since max AM allocated is .5 (3072 MB)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3733) DominantRC#compare() does not work as expected if cluster resource is empty
[ https://issues.apache.org/jira/browse/YARN-3733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14571658#comment-14571658 ] Wangda Tan commented on YARN-3733: -- Patch LGTM generally, will commit the patch once [~sunilg] +1. DominantRC#compare() does not work as expected if cluster resource is empty --- Key: YARN-3733 URL: https://issues.apache.org/jira/browse/YARN-3733 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager Affects Versions: 2.7.0 Environment: Suse 11 Sp3 , 2 NM , 2 RM one NM - 3 GB 6 v core Reporter: Bibin A Chundatt Assignee: Rohith Priority: Blocker Attachments: 0001-YARN-3733.patch, 0002-YARN-3733.patch, YARN-3733.patch Steps to reproduce = 1. Install HA with 2 RM 2 NM (3072 MB * 2 total cluster) 2. Configure map and reduce size to 512 MB after changing scheduler minimum size to 512 MB 3. Configure capacity scheduler and AM limit to .5 (DominantResourceCalculator is configured) 4. Submit 30 concurrent task 5. Switch RM Actual = For 12 Jobs AM gets allocated and all 12 starts running No other Yarn child is initiated , *all 12 Jobs in Running state for ever* Expected === Only 6 should be running at a time since max AM allocated is .5 (3072 MB) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3733) DominantRC#compare() does not work as expected if cluster resource is empty
[
https://issues.apache.org/jira/browse/YARN-3733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14570950#comment-14570950
]
Hadoop QA commented on YARN-3733:
-
\\
\\
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 17m 20s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 2 new or modified test files. |
| {color:green}+1{color} | javac | 7m 34s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 37s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 23s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 1m 40s | There were no new checkstyle
issues. |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 32s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 3m 0s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | yarn tests | 1m 57s | Tests passed in
hadoop-yarn-common. |
| {color:green}+1{color} | yarn tests | 50m 12s | Tests passed in
hadoop-yarn-server-resourcemanager. |
| | | 93m 53s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12737171/0002-YARN-3733.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / c59e745 |
| hadoop-yarn-common test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8174/artifact/patchprocess/testrun_hadoop-yarn-common.txt
|
| hadoop-yarn-server-resourcemanager test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8174/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/8174/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/8174/console |
This message was automatically generated.
DominantRC#compare() does not work as expected if cluster resource is empty
---
Key: YARN-3733
URL: https://issues.apache.org/jira/browse/YARN-3733
Project: Hadoop YARN
Issue Type: Bug
Components: resourcemanager
Affects Versions: 2.7.0
Environment: Suse 11 Sp3 , 2 NM , 2 RM
one NM - 3 GB 6 v core
Reporter: Bibin A Chundatt
Assignee: Rohith
Priority: Blocker
Attachments: 0001-YARN-3733.patch, 0002-YARN-3733.patch,
YARN-3733.patch
Steps to reproduce
=
1. Install HA with 2 RM 2 NM (3072 MB * 2 total cluster)
2. Configure map and reduce size to 512 MB after changing scheduler minimum
size to 512 MB
3. Configure capacity scheduler and AM limit to .5
(DominantResourceCalculator is configured)
4. Submit 30 concurrent task
5. Switch RM
Actual
=
For 12 Jobs AM gets allocated and all 12 starts running
No other Yarn child is initiated , *all 12 Jobs in Running state for ever*
Expected
===
Only 6 should be running at a time since max AM allocated is .5 (3072 MB)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3733) DominantRC#compare() does not work as expected if cluster resource is empty
[
https://issues.apache.org/jira/browse/YARN-3733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14570937#comment-14570937
]
Sunil G commented on YARN-3733:
---
Thank you [~rohithsharma] for the detailed information and patch.
1. Could we add a test case where only memory or vcores are more in
TestCapacityScheduler.
{code}
Resource amResource2 =
Resource.newInstance(amResourceLimit.getMemory() + 1,
amResourceLimit.getVirtualCores());
{code}
2. In TestCapacityScheduler#verifyAMLimitForLeafQueue, while submitting second
app, you could change the app name to app-2.
DominantRC#compare() does not work as expected if cluster resource is empty
---
Key: YARN-3733
URL: https://issues.apache.org/jira/browse/YARN-3733
Project: Hadoop YARN
Issue Type: Bug
Components: resourcemanager
Affects Versions: 2.7.0
Environment: Suse 11 Sp3 , 2 NM , 2 RM
one NM - 3 GB 6 v core
Reporter: Bibin A Chundatt
Assignee: Rohith
Priority: Blocker
Attachments: 0001-YARN-3733.patch, 0002-YARN-3733.patch,
YARN-3733.patch
Steps to reproduce
=
1. Install HA with 2 RM 2 NM (3072 MB * 2 total cluster)
2. Configure map and reduce size to 512 MB after changing scheduler minimum
size to 512 MB
3. Configure capacity scheduler and AM limit to .5
(DominantResourceCalculator is configured)
4. Submit 30 concurrent task
5. Switch RM
Actual
=
For 12 Jobs AM gets allocated and all 12 starts running
No other Yarn child is initiated , *all 12 Jobs in Running state for ever*
Expected
===
Only 6 should be running at a time since max AM allocated is .5 (3072 MB)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3733) DominantRC#compare() does not work as expected if cluster resource is empty
[
https://issues.apache.org/jira/browse/YARN-3733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14572085#comment-14572085
]
Rohith commented on YARN-3733:
--
bq. only memory or vcores are more in TestCapacityScheduler.
All the combination of inputs are verified in the TestResourceCalculator. And
in TestCapacityScheduler, app submission happens only for memory in
{{MockRM.submitApp}}, so default vcore minimum allocation is 1 which will be
taken by default. So just changing memory to {{amResourceLimit.getMemory() +
2}} should enough.
bq. TestCapacityScheduler#verifyAMLimitForLeafQueue, while submitting second
app, you could change the app name to app-2.
Agree.
I will upload a patch soon
DominantRC#compare() does not work as expected if cluster resource is empty
---
Key: YARN-3733
URL: https://issues.apache.org/jira/browse/YARN-3733
Project: Hadoop YARN
Issue Type: Bug
Components: resourcemanager
Affects Versions: 2.7.0
Environment: Suse 11 Sp3 , 2 NM , 2 RM
one NM - 3 GB 6 v core
Reporter: Bibin A Chundatt
Assignee: Rohith
Priority: Blocker
Attachments: 0001-YARN-3733.patch, 0002-YARN-3733.patch,
YARN-3733.patch
Steps to reproduce
=
1. Install HA with 2 RM 2 NM (3072 MB * 2 total cluster)
2. Configure map and reduce size to 512 MB after changing scheduler minimum
size to 512 MB
3. Configure capacity scheduler and AM limit to .5
(DominantResourceCalculator is configured)
4. Submit 30 concurrent task
5. Switch RM
Actual
=
For 12 Jobs AM gets allocated and all 12 starts running
No other Yarn child is initiated , *all 12 Jobs in Running state for ever*
Expected
===
Only 6 should be running at a time since max AM allocated is .5 (3072 MB)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3733) DominantRC#compare() does not work as expected if cluster resource is empty
[ https://issues.apache.org/jira/browse/YARN-3733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14568682#comment-14568682 ] Rohith commented on YARN-3733: -- Updated the summary as per defect. DominantRC#compare() does not work as expected if cluster resource is empty --- Key: YARN-3733 URL: https://issues.apache.org/jira/browse/YARN-3733 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager Affects Versions: 2.7.0 Environment: Suse 11 Sp3 , 2 NM , 2 RM one NM - 3 GB 6 v core Reporter: Bibin A Chundatt Assignee: Rohith Priority: Blocker Attachments: YARN-3733.patch Steps to reproduce = 1. Install HA with 2 RM 2 NM (3072 MB * 2 total cluster) 2. Configure map and reduce size to 512 MB after changing scheduler minimum size to 512 MB 3. Configure capacity scheduler and AM limit to .5 (DominantResourceCalculator is configured) 4. Submit 30 concurrent task 5. Switch RM Actual = For 12 Jobs AM gets allocated and all 12 starts running No other Yarn child is initiated , *all 12 Jobs in Running state for ever* Expected === Only 6 should be running at a time since max AM allocated is .5 (3072 MB) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3733) DominantRC#compare() does not work as expected if cluster resource is empty
[
https://issues.apache.org/jira/browse/YARN-3733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14570025#comment-14570025
]
Wangda Tan commented on YARN-3733:
--
Took a look at the patch and discussion. Thanks for working on this
[~rohithsharma].
I think [~sunilg] mentioned
https://issues.apache.org/jira/browse/YARN-3733?focusedCommentId=14568880page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14568880
makes sense to me. If the clusterResource is 0, we can compare individual
resource type. It could be:
{code}
Returns : when l.mem right.mem || l.cpu right.cpu
Returns =: when (l.mem = right.mem l.cpu = right.cpu) || (l.mem =
right.mem l.cpu = right.cpu)
Returns : when l.mem right.mem || l.cpu right.cpu
{code}
This produces same result as the INF approach in the patch, but also can
compare if both l/r have 0 values. The reason I prefer this is, I'm sure the
patch can solve the am-resource-percent problem. But with suggested approach,
we can make sure getting more reasonable result if we need to compare
non-zero-resource when clusterResource is zero. (For example, sort applications
by their requirements when clusterResource is zero).
And to avoid future regression, could you add a test to verify the
am-resource-limit problem is solved?
DominantRC#compare() does not work as expected if cluster resource is empty
---
Key: YARN-3733
URL: https://issues.apache.org/jira/browse/YARN-3733
Project: Hadoop YARN
Issue Type: Bug
Components: resourcemanager
Affects Versions: 2.7.0
Environment: Suse 11 Sp3 , 2 NM , 2 RM
one NM - 3 GB 6 v core
Reporter: Bibin A Chundatt
Assignee: Rohith
Priority: Blocker
Attachments: 0001-YARN-3733.patch, YARN-3733.patch
Steps to reproduce
=
1. Install HA with 2 RM 2 NM (3072 MB * 2 total cluster)
2. Configure map and reduce size to 512 MB after changing scheduler minimum
size to 512 MB
3. Configure capacity scheduler and AM limit to .5
(DominantResourceCalculator is configured)
4. Submit 30 concurrent task
5. Switch RM
Actual
=
For 12 Jobs AM gets allocated and all 12 starts running
No other Yarn child is initiated , *all 12 Jobs in Running state for ever*
Expected
===
Only 6 should be running at a time since max AM allocated is .5 (3072 MB)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3733) DominantRC#compare() does not work as expected if cluster resource is empty
[
https://issues.apache.org/jira/browse/YARN-3733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14568913#comment-14568913
]
Hadoop QA commented on YARN-3733:
-
\\
\\
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 16m 6s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:green}+1{color} | javac | 7m 33s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 36s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 22s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 0m 54s | There were no new checkstyle
issues. |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 33s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 32s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 1m 33s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | yarn tests | 1m 57s | Tests passed in
hadoop-yarn-common. |
| | | 40m 10s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12736802/0001-YARN-3733.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 990078b |
| hadoop-yarn-common test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8166/artifact/patchprocess/testrun_hadoop-yarn-common.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/8166/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/8166/console |
This message was automatically generated.
DominantRC#compare() does not work as expected if cluster resource is empty
---
Key: YARN-3733
URL: https://issues.apache.org/jira/browse/YARN-3733
Project: Hadoop YARN
Issue Type: Bug
Components: resourcemanager
Affects Versions: 2.7.0
Environment: Suse 11 Sp3 , 2 NM , 2 RM
one NM - 3 GB 6 v core
Reporter: Bibin A Chundatt
Assignee: Rohith
Priority: Blocker
Attachments: 0001-YARN-3733.patch, YARN-3733.patch
Steps to reproduce
=
1. Install HA with 2 RM 2 NM (3072 MB * 2 total cluster)
2. Configure map and reduce size to 512 MB after changing scheduler minimum
size to 512 MB
3. Configure capacity scheduler and AM limit to .5
(DominantResourceCalculator is configured)
4. Submit 30 concurrent task
5. Switch RM
Actual
=
For 12 Jobs AM gets allocated and all 12 starts running
No other Yarn child is initiated , *all 12 Jobs in Running state for ever*
Expected
===
Only 6 should be running at a time since max AM allocated is .5 (3072 MB)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3733) DominantRC#compare() does not work as expected if cluster resource is empty
[ https://issues.apache.org/jira/browse/YARN-3733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14568880#comment-14568880 ] Sunil G commented on YARN-3733: --- Hi [~rohithsharma] Thanks for the detailed scenario. Scenario 4 can be possible, correct?. clusterResource0,0 : lhs 2,2 and rhs 3,2. Currently getResourceAsValue gives back the max ratio of mem/vcores if dominent. Else gives the min ratio. If clusterResource is 0, then could we directly send the max of mem/vcore if dominent, and min in other case. This has to be made more better algorithm when more resources comes in. This is not completely perfect as we treat memory and vcores leniently. Pls share your thoughts. DominantRC#compare() does not work as expected if cluster resource is empty --- Key: YARN-3733 URL: https://issues.apache.org/jira/browse/YARN-3733 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager Affects Versions: 2.7.0 Environment: Suse 11 Sp3 , 2 NM , 2 RM one NM - 3 GB 6 v core Reporter: Bibin A Chundatt Assignee: Rohith Priority: Blocker Attachments: 0001-YARN-3733.patch, YARN-3733.patch Steps to reproduce = 1. Install HA with 2 RM 2 NM (3072 MB * 2 total cluster) 2. Configure map and reduce size to 512 MB after changing scheduler minimum size to 512 MB 3. Configure capacity scheduler and AM limit to .5 (DominantResourceCalculator is configured) 4. Submit 30 concurrent task 5. Switch RM Actual = For 12 Jobs AM gets allocated and all 12 starts running No other Yarn child is initiated , *all 12 Jobs in Running state for ever* Expected === Only 6 should be running at a time since max AM allocated is .5 (3072 MB) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
