[jira] [Commented] (YARN-3136) getTransferredContainers can be a bottleneck during AM registration
[ https://issues.apache.org/jira/browse/YARN-3136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16303440#comment-16303440 ] stefanlee commented on YARN-3136: - [~jlowe] thanks for this jira, could you please tell me what is the jira about *We've already done similar work during AM allocate calls to make sure they don't needlessly grab the scheduler lock* ? i found *AM allocate* BLOCKED easily in my RM. > getTransferredContainers can be a bottleneck during AM registration > --- > > Key: YARN-3136 > URL: https://issues.apache.org/jira/browse/YARN-3136 > Project: Hadoop YARN > Issue Type: Sub-task > Components: scheduler >Affects Versions: 2.6.0 >Reporter: Jason Lowe >Assignee: Sunil G > Labels: 2.7.2-candidate > Fix For: 2.8.0, 2.7.2, 3.0.0-alpha1 > > Attachments: 0001-YARN-3136.patch, 00010-YARN-3136.patch, > 00011-YARN-3136.patch, 00012-YARN-3136.patch, 00013-YARN-3136.patch, > 0002-YARN-3136.patch, 0003-YARN-3136.patch, 0004-YARN-3136.patch, > 0005-YARN-3136.patch, 0006-YARN-3136.patch, 0007-YARN-3136.patch, > 0008-YARN-3136.patch, 0009-YARN-3136.patch, YARN-3136.branch-2.7.patch > > > While examining RM stack traces on a busy cluster I noticed a pattern of AMs > stuck waiting for the scheduler lock trying to call getTransferredContainers. > The scheduler lock is highly contended, especially on a large cluster with > many nodes heartbeating, and it would be nice if we could find a way to > eliminate the need to grab this lock during this call. We've already done > similar work during AM allocate calls to make sure they don't needlessly grab > the scheduler lock, and it would be good to do so here as well, if possible. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6972) Adding RM ClusterId in AppInfo
[
https://issues.apache.org/jira/browse/YARN-6972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16303372#comment-16303372
]
genericqa commented on YARN-6972:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 17m
26s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 25m
7s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
5s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
11s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 41s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
9s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
48s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
16m 25s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}101m 49s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}190m 24s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestQueueState |
| | hadoop.yarn.server.resourcemanager.webapp.TestRMWebServiceAppsNodelabel |
| |
hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesAppsModification |
| |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestIncreaseAllocationExpirer
|
| |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerResizing |
| | hadoop.yarn.server.resourcemanager.TestWorkPreservingRMRestart |
| |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacitySchedulerWithMultiResourceTypes
|
| |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestNodeLabelContainerAllocation
|
| |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestQueueManagementDynamicEditPolicy
|
| | hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesApps |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | YARN-6972 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12903614/YARN-6972.009.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux eb98cb07e148 3.13.0-129-generic
[jira] [Commented] (YARN-7601) Incorrect container states recovered as LevelDB uses alphabetical order
[
https://issues.apache.org/jira/browse/YARN-7601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16303351#comment-16303351
]
genericqa commented on YARN-7601:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 17m
38s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 25m
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
34s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
1s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 5s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
41s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
16m 47s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 22m 11s{color}
| {color:red} hadoop-yarn-server-nodemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
54s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}110m 17s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.nodemanager.containermanager.scheduler.TestContainerSchedulerQueuing
|
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | YARN-7601 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12903640/YARN-7601.001.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 573403f52efa 3.13.0-129-generic #178-Ubuntu SMP Fri Aug 11
12:48:20 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 52babbb |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-YARN-Build/19027/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/19027/testReport/ |
| Max. process+thread count | 334 (vs. ulimit of 5000) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
U:
[jira] [Created] (YARN-7682) Expose canAssign method in the PlacementConstraintManager
Arun Suresh created YARN-7682:
-
Summary: Expose canAssign method in the PlacementConstraintManager
Key: YARN-7682
URL: https://issues.apache.org/jira/browse/YARN-7682
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Arun Suresh
Assignee: Panagiotis Garefalakis
As per discussion in YARN-7613. Lets expose {{canAssign}} method in the
PlacementConstraintManager that takes a sourceTags, applicationId,
SchedulerNode and AllocationTagsManager and returns true if constraints are not
violated by placing the container on the node.
I prefer not passing in the SchedulingRequest, since it can have > 1
numAllocations. We want this api to be called for single allocations.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7613) Implement Planning algorithms for rich placement
[
https://issues.apache.org/jira/browse/YARN-7613?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16303343#comment-16303343
]
Arun Suresh commented on YARN-7613:
---
bq. in a previous draft patch I had the method as part of the constraint
manager for example which did make some sense.
Yeah, that's probably a good idea - since the PCM is available to the scheduler
anyway (while the algorithm is not). We can just call:
{code}
public boolean attemptAllocationOnNode(SchedulerApplicationAttempt appAttempt,
SchedulingRequest schedulingRequest, SchedulerNode schedulerNode) {
...
PlacementConstraintManager pcm = ...
AllocationTagsManager tm = ...
if (!pcm.canAssign(appAttempt.getApplicationId(), schedulingRequest,
schedulerNode, tm))) {
return false;
}
...
{code}
w.r.t the AlgorithmContext
Yeah, that should still be something only the Dispatcher and Algorithm
interacts with - an argument to the place() method. Lets keep that out of the
Scheduler phase.
We should have the canAssign change as its own JIRA and have YARN-7681 and this
depend on it.
> Implement Planning algorithms for rich placement
>
>
> Key: YARN-7613
> URL: https://issues.apache.org/jira/browse/YARN-7613
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Arun Suresh
>Assignee: Panagiotis Garefalakis
> Attachments: YARN-7613.wip.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-7613) Implement Planning algorithms for rich placement
[ https://issues.apache.org/jira/browse/YARN-7613?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16303327#comment-16303327 ] Panagiotis Garefalakis edited comment on YARN-7613 at 12/25/17 5:29 PM: Thanks for the comments [~asuresh] bq. Based on discussions in YARN-7612, Lets move the canAssign method into a separate utility class and handle that in YARN-7681. It should definitely be in a separate class but I am not sure it should be a utility one. It is part of constraint satisfaction so it looks quite fundamental to me - in a previous draft patch I had the method as part of the constraint manager for example which did make some sense. bq. At the very least, it should take the container source tags (from the SchedulingRequest), the target SchedulerNode, the TagsManager and the ConstraintsManager - and return true if it can assign the source tag to the Scheduler node without violating any constraints. ApplicationId is also needed since we query the tags based on that but I agree about the rest - would also pass the SchedulingRequest object instead of tags as they contain more information. AlgorithmContext (described in your previous comment) could be helpful here since we would not have to pass Tags and Constraints Managers explicitly. was (Author: pgaref): Thanks for the comments [~asuresh] bq. Based on discussions in YARN-7612, Lets move the canAssign method into a separate utility class and handle that in YARN-7681. It should definitely be in a separate class but I am not sure it should be a utility one. It is part of constraint satisfaction so it looks quite fundamental to me - in a previous draft patch I had the method as part of the constraint manager for example which did make some sense. bq. At the very least, it should take the container source tags (from the SchedulingRequest), the target SchedulerNode, the TagsManager and the ConstraintsManager - and return true if it can assign the source tag to the Scheduler node without violating any constraints. ApplicationId is also needed since we query the tags based on that but I agree about the rest - would also pass the SchedulingRequest object instead of tags as they contain more information. AlgorithmContext (described in your previous comment) could be helpful in this as well since we would not have to pass Tags and Constraints Managers explicitly. > Implement Planning algorithms for rich placement > > > Key: YARN-7613 > URL: https://issues.apache.org/jira/browse/YARN-7613 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Arun Suresh >Assignee: Panagiotis Garefalakis > Attachments: YARN-7613.wip.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-7613) Implement Planning algorithms for rich placement
[ https://issues.apache.org/jira/browse/YARN-7613?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16303327#comment-16303327 ] Panagiotis Garefalakis edited comment on YARN-7613 at 12/25/17 5:28 PM: Thanks for the comments [~asuresh] bq. Based on discussions in YARN-7612, Lets move the canAssign method into a separate utility class and handle that in YARN-7681. It should definitely be in a separate class but I am not sure it should be a utility one. It is part of constraint satisfaction so it looks quite fundamental to me - in a previous draft patch I had the method as part of the constraint manager for example which did make some sense. bq. At the very least, it should take the container source tags (from the SchedulingRequest), the target SchedulerNode, the TagsManager and the ConstraintsManager - and return true if it can assign the source tag to the Scheduler node without violating any constraints. ApplicationId is also needed since we query the tags based on that but I agree about the rest - would also pass the SchedulingRequest object instead of tags as they contain more information. AlgorithmContext (described in your previous comment) could be helpful in this as well since we would not have to pass Tags and Constraints Managers explicitly. was (Author: pgaref): bq. Based on discussions in YARN-7612, Lets move the canAssign method into a separate utility class and handle that in YARN-7681. It should definitely be in a separate class but I am not sure it should be a utility one. It is part of constraint satisfaction so it looks quite fundamental to me - in a previous draft patch I had the method as part of the constraint manager for example which did make some sense. bq. At the very least, it should take the container source tags (from the SchedulingRequest), the target SchedulerNode, the TagsManager and the ConstraintsManager - and return true if it can assign the source tag to the Scheduler node without violating any constraints. ApplicationId is also needed since we query the tags based on that but I agree about the rest - would also pass the SchedulingRequest object instead of tags as they contain more information. AlgorithmContext (described in your previous comment) could be helpful in this as well since we would not have to pass Tags and Constraints Managers explicitly. > Implement Planning algorithms for rich placement > > > Key: YARN-7613 > URL: https://issues.apache.org/jira/browse/YARN-7613 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Arun Suresh >Assignee: Panagiotis Garefalakis > Attachments: YARN-7613.wip.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7613) Implement Planning algorithms for rich placement
[ https://issues.apache.org/jira/browse/YARN-7613?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16303327#comment-16303327 ] Panagiotis Garefalakis commented on YARN-7613: -- bq. Based on discussions in YARN-7612, Lets move the canAssign method into a separate utility class and handle that in YARN-7681. It should definitely be in a separate class but I am not sure it should be a utility one. It is part of constraint satisfaction so it looks quite fundamental to me - in a previous draft patch I had the method as part of the constraint manager for example which did make some sense. bq. At the very least, it should take the container source tags (from the SchedulingRequest), the target SchedulerNode, the TagsManager and the ConstraintsManager - and return true if it can assign the source tag to the Scheduler node without violating any constraints. ApplicationId is also needed since we query the tags based on that but I agree about the rest - would also pass the SchedulingRequest object instead of tags as they contain more information. AlgorithmContext (described in your previous comment) could be helpful in this as well since we would not have to pass Tags and Constraints Managers explicitly. > Implement Planning algorithms for rich placement > > > Key: YARN-7613 > URL: https://issues.apache.org/jira/browse/YARN-7613 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Arun Suresh >Assignee: Panagiotis Garefalakis > Attachments: YARN-7613.wip.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7613) Implement Planning algorithms for rich placement
[
https://issues.apache.org/jira/browse/YARN-7613?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16303320#comment-16303320
]
Arun Suresh commented on YARN-7613:
---
Based on discussions in YARN-7612, Lets move the {{canAssign}} method into a
separate utility class and handle that in YARN-7681.
At the very least, it should take the container source tags (from the
SchedulingRequest), the target SchedulerNode, the TagsManager and the
ConstraintsManager - and return true if it can assign the source tag to the
Scheduler node without violating any constraints.
Thoughts ?
> Implement Planning algorithms for rich placement
>
>
> Key: YARN-7613
> URL: https://issues.apache.org/jira/browse/YARN-7613
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Arun Suresh
>Assignee: Panagiotis Garefalakis
> Attachments: YARN-7613.wip.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7601) Incorrect container states recovered as LevelDB uses alphabetical order
[ https://issues.apache.org/jira/browse/YARN-7601?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sampada Dehankar updated YARN-7601: --- Attachment: YARN-7601.001.patch > Incorrect container states recovered as LevelDB uses alphabetical order > --- > > Key: YARN-7601 > URL: https://issues.apache.org/jira/browse/YARN-7601 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Sampada Dehankar >Assignee: Sampada Dehankar > Attachments: YARN-7601.001.patch > > > LevelDB stores key-value pairs in the alphabetical order. Container id > concatenated by its state is used as key. So, even if container goes through > any states in its life cycle, the order of states for following values > retrieved from LevelDB is always going to be as below`: > LAUNCHED > PAUSED > QUEUED > For eg: If a container is LAUNCHED then PAUSED and LAUNCHED again, the > recovered container state is PAUSED currently instead of LAUNCHED. > We propose to store the timestamp as the value while making call to > > storeContainerLaunched > storeContainerPaused > storeContainerQueued > > so that correct container state is recovered based on timestamps. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7612) Add Processor Framework for Rich Placement Constraints
[ https://issues.apache.org/jira/browse/YARN-7612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16303265#comment-16303265 ] Weiwei Yang commented on YARN-7612: --- Thanks [~asuresh], I have created YARN-7681 to followup on this. Lets move the discussion to that one. > Add Processor Framework for Rich Placement Constraints > -- > > Key: YARN-7612 > URL: https://issues.apache.org/jira/browse/YARN-7612 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Arun Suresh >Assignee: Arun Suresh > Fix For: 3.1.0 > > Attachments: YARN-7612-YARN-6592.001.patch, > YARN-7612-YARN-6592.002.patch, YARN-7612-YARN-6592.003.patch, > YARN-7612-YARN-6592.004.patch, YARN-7612-YARN-6592.005.patch, > YARN-7612-YARN-6592.006.patch, YARN-7612-YARN-6592.007.patch, > YARN-7612-YARN-6592.008.patch, YARN-7612-YARN-6592.009.patch, > YARN-7612-YARN-6592.010.patch, YARN-7612-YARN-6592.011.patch, > YARN-7612-YARN-6592.012.patch, YARN-7612-v2.wip.patch, YARN-7612.wip.patch > > > This introduces a Placement Processor and a Planning algorithm framework to > handle placement constraints and scheduling requests from an app and places > them on nodes. > The actual planning algorithm(s) will be handled in a YARN-7613. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-7681) Scheduler should double-check placement constraint before actual allocation is made
Weiwei Yang created YARN-7681: - Summary: Scheduler should double-check placement constraint before actual allocation is made Key: YARN-7681 URL: https://issues.apache.org/jira/browse/YARN-7681 Project: Hadoop YARN Issue Type: Sub-task Components: RM, scheduler Reporter: Weiwei Yang Assignee: Weiwei Yang This JIRA is created based on the discussions under YARN-7612, see comments after [this comment|https://issues.apache.org/jira/browse/YARN-7612?focusedCommentId=16303051=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16303051]. AllocationTagsManager maintains tags info that helps to make placement decisions at placement phase, however tags are changing along with container's lifecycle, so it is possible that the placement violates the constraints at the scheduling phase. Propose to add an extra check in the scheduler to make sure constraints are still satisfied during the actual allocation. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4980) SchedulerEventDispatcher should not process by only a single EventProcessor thread
[ https://issues.apache.org/jira/browse/YARN-4980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16303252#comment-16303252 ] stefanlee commented on YARN-4980: - [~chenfolin] hi,have you resolved this problem, or optimized it? > SchedulerEventDispatcher should not process by only a single EventProcessor > thread > -- > > Key: YARN-4980 > URL: https://issues.apache.org/jira/browse/YARN-4980 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 2.5.0, 2.6.4 > Environment: 1 resourcemanager > 500 nodemanager > nodemanager heartbeat interval is 3 secs. >Reporter: ChenFolin > > I think only a single EventProcessor thread in SchedulerEventDispatcher to > process event is unreasonable. > I often see "Size of scheduler event-queue is 1000(2000..)..." at the > resourcemanager log. > Now I have 500 nodemanager , event process is slowly. If I add another 200 > nodemanagers , the situation will badly. > ... so,can node events(node add/delete/update...) processed by multi threads? -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
