[jira] [Commented] (YARN-5987) NM configured command to collect heap dump of preempted container
[
https://issues.apache.org/jira/browse/YARN-5987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15941551#comment-15941551
]
Hadoop QA commented on YARN-5987:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
20s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
46s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m
7s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
57s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
40s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 10m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
3s{color} | {color:green} hadoop-yarn-project/hadoop-yarn: The patch generated
0 new + 433 unchanged - 46 fixed = 433 total (was 479) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
36s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
43s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 14m
5s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
32s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 83m 58s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | YARN-5987 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12843500/YARN-5987.001.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle xml |
| uname | Linux aafedca96031 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6
15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 84ddedc |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/15387/testReport/ |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
[jira] [Commented] (YARN-6363) Extending SLS: Synthetic Load Generator
[
https://issues.apache.org/jira/browse/YARN-6363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15941528#comment-15941528
]
Hadoop QA commented on YARN-6363:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
14s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 9 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
42s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 12m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 16m
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
58s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
2s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
19s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
13s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
15s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 13m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 13m
47s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
2m 3s{color} | {color:orange} root: The patch generated 196 new + 271
unchanged - 21 fixed = 467 total (was 292) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
12s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch 1 line(s) with tabs. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
4s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
47s{color} | {color:red} hadoop-tools/hadoop-sls generated 1 new + 0 unchanged
- 0 fixed = 1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
23s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 39m 47s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
31s{color} | {color:green} hadoop-rumen in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
35s{color} | {color:green} hadoop-sls in the patch passed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
40s{color} | {color:red} The patch generated 2 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black}129m 58s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-tools/hadoop-sls |
| | org.apache.hadoop.yarn.sls.SLSRunner.run(String[]) invokes
System.exit(...), which shuts down the entire virtual machine At
SLSRunner.java:down the entire virtual machine At SLSRunner.java:[line 741] |
| Failed junit tests | hadoop.yarn.server.resourcemanager.TestRMRestart |
| |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacitySchedulerDynamicBehavior
|
| | hadoop.yarn.server.resourcemanager.reservation.TestReservationSystem |
| Timed out junit tests |
org.apache.hadoop.yarn.server.resourcemanager.TestReservationSystemWithRMHA |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | YARN-6363 |
| JIRA Patch URL |
[jira] [Assigned] (YARN-5987) NM configured command to collect heap dump of preempted container
[ https://issues.apache.org/jira/browse/YARN-5987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Szegedi reassigned YARN-5987: Assignee: (was: Miklos Szegedi) > NM configured command to collect heap dump of preempted container > - > > Key: YARN-5987 > URL: https://issues.apache.org/jira/browse/YARN-5987 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Miklos Szegedi > Attachments: YARN-5987.000.patch, YARN-5987.001.patch > > > The node manager can kill a container, if it exceeds the assigned memory > limits. It would be nice to have a configuration entry to set up a command > that can collect additional debug information, if needed. The collected > information can be used for root cause analysis. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6302) Fail the node, if Linux Container Executor is not configured properly
[
https://issues.apache.org/jira/browse/YARN-6302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15941518#comment-15941518
]
ASF GitHub Bot commented on YARN-6302:
--
Github user szegedim commented on a diff in the pull request:
https://github.com/apache/hadoop/pull/200#discussion_r108025819
--- Diff:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/LinuxContainerExecutor.java
---
@@ -551,16 +550,16 @@ public int launchContainer(ContainerStartContext ctx)
} else {
LOG.info(
"Container was marked as inactive. Returning terminated
error");
-return ExitCode.TERMINATED.getExitCode();
+return ContainerExecutor.ExitCode.TERMINATED.getExitCode();
--- End diff --
There is one ExitCode now in this class as well.
> Fail the node, if Linux Container Executor is not configured properly
> -
>
> Key: YARN-6302
> URL: https://issues.apache.org/jira/browse/YARN-6302
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Miklos Szegedi
>Assignee: Miklos Szegedi
>Priority: Minor
>
> We have a cluster that has one node with misconfigured Linux Container
> Executor. Every time an AM or regular container is launched on the cluster,
> it will fail. The node will still have resources available, so it keeps
> failing apps until the administrator notices the issue and decommissions the
> node. AM Blacklisting only helps, if the application is already running.
> As a possible improvement, when the LCE is used on the cluster and a NM gets
> certain errors back from the LCE, like error 24 configuration not found, we
> should not try to allocate anything on the node anymore or shut down the node
> entirely. That kind of problem normally does not fix itself and it means that
> nothing can really run on that node.
> {code}
> Application application_1488920587909_0010 failed 2 times due to AM Container
> for appattempt_1488920587909_0010_02 exited with exitCode: -1000
> Failing this attempt.Diagnostics: Application application_1488920587909_0010
> initialization failed (exitCode=24) with output:
> For more detailed output, check the application tracking page:
> http://node-1.domain.com:8088/cluster/app/application_1488920587909_0010 Then
> click on links to logs of each attempt.
> . Failing the application.
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6302) Fail the node, if Linux Container Executor is not configured properly
[
https://issues.apache.org/jira/browse/YARN-6302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15941517#comment-15941517
]
ASF GitHub Bot commented on YARN-6302:
--
Github user szegedim commented on a diff in the pull request:
https://github.com/apache/hadoop/pull/200#discussion_r108025814
--- Diff:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/LinuxContainerExecutor.java
---
@@ -580,19 +579,19 @@ public int launchContainer(ContainerStartContext ctx)
logOutput(diagnostics);
container.handle(new ContainerDiagnosticsUpdateEvent(containerId,
diagnostics));
-if (exitCode == LinuxContainerExecutorExitCode.
+if (exitCode == ExitCode.
--- End diff --
Done.
> Fail the node, if Linux Container Executor is not configured properly
> -
>
> Key: YARN-6302
> URL: https://issues.apache.org/jira/browse/YARN-6302
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Miklos Szegedi
>Assignee: Miklos Szegedi
>Priority: Minor
>
> We have a cluster that has one node with misconfigured Linux Container
> Executor. Every time an AM or regular container is launched on the cluster,
> it will fail. The node will still have resources available, so it keeps
> failing apps until the administrator notices the issue and decommissions the
> node. AM Blacklisting only helps, if the application is already running.
> As a possible improvement, when the LCE is used on the cluster and a NM gets
> certain errors back from the LCE, like error 24 configuration not found, we
> should not try to allocate anything on the node anymore or shut down the node
> entirely. That kind of problem normally does not fix itself and it means that
> nothing can really run on that node.
> {code}
> Application application_1488920587909_0010 failed 2 times due to AM Container
> for appattempt_1488920587909_0010_02 exited with exitCode: -1000
> Failing this attempt.Diagnostics: Application application_1488920587909_0010
> initialization failed (exitCode=24) with output:
> For more detailed output, check the application tracking page:
> http://node-1.domain.com:8088/cluster/app/application_1488920587909_0010 Then
> click on links to logs of each attempt.
> . Failing the application.
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5829) FS preemption should reserve a node before considering containers on it for preemption
[
https://issues.apache.org/jira/browse/YARN-5829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15941504#comment-15941504
]
ASF GitHub Bot commented on YARN-5829:
--
Github user szegedim commented on a diff in the pull request:
https://github.com/apache/hadoop/pull/201#discussion_r108025378
--- Diff:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairSchedulerPreemption.java
---
@@ -294,11 +294,29 @@ private void verifyPreemption(int
numStarvedAppContainers)
8 - 2 * numStarvedAppContainers,
greedyApp.getQueue().getMetrics().getAggregatePreemptedContainers());
+// Verify the node is reserved for the starvingApp
+for (RMNode rmNode : rmNodes) {
+ FSSchedulerNode node = (FSSchedulerNode)
+ scheduler.getNodeTracker().getNode(rmNode.getNodeID());
+ if (node.getContainersForPreemption().size() > 0) {
+
assertTrue(node.getPreemptionList().keySet().contains(starvingApp));
+ }
+}
+
sendEnoughNodeUpdatesToAssignFully();
// Verify the preempted containers are assigned to starvingApp
assertEquals("Starved app is not assigned the right # of containers",
numStarvedAppContainers, starvingApp.getLiveContainers().size());
+
+// Verify the node is not reserved for the starvingApp anymore
+for (RMNode rmNode : rmNodes) {
+ FSSchedulerNode node = (FSSchedulerNode)
+ scheduler.getNodeTracker().getNode(rmNode.getNodeID());
+ if (node.getContainersForPreemption().size() > 0) {
+
assertTrue(!node.getPreemptionList().keySet().contains(starvingApp));
--- End diff --
Done.
> FS preemption should reserve a node before considering containers on it for
> preemption
> --
>
> Key: YARN-5829
> URL: https://issues.apache.org/jira/browse/YARN-5829
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: fairscheduler
>Reporter: Karthik Kambatla
>Assignee: Miklos Szegedi
>
> FS preemption evaluates nodes for preemption, and subsequently preempts
> identified containers. If this node is not reserved for a specific
> application, any other application could be allocated resources on this node.
> Reserving the node for the starved application before preempting containers
> would help avoid this.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5829) FS preemption should reserve a node before considering containers on it for preemption
[
https://issues.apache.org/jira/browse/YARN-5829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15941503#comment-15941503
]
ASF GitHub Bot commented on YARN-5829:
--
Github user szegedim commented on a diff in the pull request:
https://github.com/apache/hadoop/pull/201#discussion_r108025370
--- Diff:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairSchedulerPreemption.java
---
@@ -294,11 +294,29 @@ private void verifyPreemption(int
numStarvedAppContainers)
8 - 2 * numStarvedAppContainers,
greedyApp.getQueue().getMetrics().getAggregatePreemptedContainers());
+// Verify the node is reserved for the starvingApp
+for (RMNode rmNode : rmNodes) {
+ FSSchedulerNode node = (FSSchedulerNode)
+ scheduler.getNodeTracker().getNode(rmNode.getNodeID());
+ if (node.getContainersForPreemption().size() > 0) {
+
assertTrue(node.getPreemptionList().keySet().contains(starvingApp));
--- End diff --
Done.
> FS preemption should reserve a node before considering containers on it for
> preemption
> --
>
> Key: YARN-5829
> URL: https://issues.apache.org/jira/browse/YARN-5829
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: fairscheduler
>Reporter: Karthik Kambatla
>Assignee: Miklos Szegedi
>
> FS preemption evaluates nodes for preemption, and subsequently preempts
> identified containers. If this node is not reserved for a specific
> application, any other application could be allocated resources on this node.
> Reserving the node for the starved application before preempting containers
> would help avoid this.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5829) FS preemption should reserve a node before considering containers on it for preemption
[
https://issues.apache.org/jira/browse/YARN-5829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15941502#comment-15941502
]
ASF GitHub Bot commented on YARN-5829:
--
Github user szegedim commented on a diff in the pull request:
https://github.com/apache/hadoop/pull/201#discussion_r108025337
--- Diff:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFSSchedulerNode.java
---
@@ -0,0 +1,376 @@
+package org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair;
+
+import org.apache.hadoop.yarn.api.records.*;
+import
org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainer;
+import org.apache.hadoop.yarn.server.resourcemanager.rmnode.RMNode;
+import org.apache.hadoop.yarn.util.resource.Resources;
+import org.junit.Test;
+import org.mockito.invocation.InvocationOnMock;
+import org.mockito.stubbing.Answer;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Map;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNotEquals;
+import static org.junit.Assert.assertTrue;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+/**
+ * Test scheduler node, especially preemption reservations.
+ */
+public class TestFSSchedulerNode {
+ private long containerNum = 0;
+ private ArrayList containers = new ArrayList<>();
+
+ private RMNode createNode() {
+RMNode node = mock(RMNode.class);
+when(node.getTotalCapability()).thenReturn(Resource.newInstance(8192,
8));
+when(node.getHostName()).thenReturn("host.domain.com");
+return node;
+ }
+
+ private RMContainer createDefaultContainer() {
+return createContainer(Resource.newInstance(1024, 1), null);
+ }
+
+ private RMContainer createContainer(
+ Resource request, ApplicationAttemptId appAttemptId) {
+RMContainer container = mock(RMContainer.class);
+Container containerInner = mock(Container.class);
+ContainerId id = mock(ContainerId.class);
+when(id.getContainerId()).thenReturn(containerNum);
+when(containerInner.getResource()).
+thenReturn(Resources.clone(request));
+when(containerInner.getId()).thenReturn(id);
+when(containerInner.getExecutionType()).
+thenReturn(ExecutionType.GUARANTEED);
+when(container.getApplicationAttemptId()).thenReturn(appAttemptId);
+when(container.getContainerId()).thenReturn(id);
+when(container.getContainer()).thenReturn(containerInner);
+
when(container.getExecutionType()).thenReturn(ExecutionType.GUARANTEED);
+when(container.getAllocatedResource()).
+thenReturn(Resources.clone(request));
+containers.add(container);
+containerNum++;
+return container;
+ }
+
+ private void saturateCluster(FSSchedulerNode schedulerNode) {
+while (!Resources.isNone(schedulerNode.getUnallocatedResource())) {
+ createDefaultContainer();
+ schedulerNode.allocateContainer(containers.get((int)containerNum -
1));
+ schedulerNode.containerStarted(containers.get((int)containerNum - 1).
+ getContainerId());
+}
+ }
+
+ private FSAppAttempt createStarvingApp(FSSchedulerNode schedulerNode,
+ Resource request) {
+FSAppAttempt starvingApp = mock(FSAppAttempt.class);
+ApplicationAttemptId appAttemptId =
+mock(ApplicationAttemptId.class);
+when(starvingApp.getApplicationAttemptId()).thenReturn(appAttemptId);
+when(starvingApp.assignContainer(schedulerNode)).thenAnswer(
+new Answer() {
+ @Override
+ public Resource answer(InvocationOnMock invocationOnMock)
+ throws Throwable {
+Resource response = Resource.newInstance(0, 0);
+while (!Resources.isNone(request) &&
+!Resources.isNone(schedulerNode.getUnallocatedResource()))
{
+ RMContainer container = createContainer(request,
appAttemptId);
+ schedulerNode.allocateContainer(container);
+ Resources.addTo(response, container.getAllocatedResource());
+ Resources.subtractFrom(request,
+ container.getAllocatedResource());
+}
+return response;
+ }
+});
+when(starvingApp.getPendingDemand()).thenReturn(request);
+return starvingApp;
+ }
+
+ private void finalValidation(FSSchedulerNode schedulerNode) {
+assertEquals("Everything should
[jira] [Commented] (YARN-5829) FS preemption should reserve a node before considering containers on it for preemption
[
https://issues.apache.org/jira/browse/YARN-5829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15941501#comment-15941501
]
ASF GitHub Bot commented on YARN-5829:
--
Github user szegedim commented on a diff in the pull request:
https://github.com/apache/hadoop/pull/201#discussion_r108025333
--- Diff:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFSSchedulerNode.java
---
@@ -0,0 +1,376 @@
+package org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair;
+
+import org.apache.hadoop.yarn.api.records.*;
+import
org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainer;
+import org.apache.hadoop.yarn.server.resourcemanager.rmnode.RMNode;
+import org.apache.hadoop.yarn.util.resource.Resources;
+import org.junit.Test;
+import org.mockito.invocation.InvocationOnMock;
+import org.mockito.stubbing.Answer;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Map;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNotEquals;
+import static org.junit.Assert.assertTrue;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+/**
+ * Test scheduler node, especially preemption reservations.
+ */
+public class TestFSSchedulerNode {
+ private long containerNum = 0;
+ private ArrayList containers = new ArrayList<>();
+
+ private RMNode createNode() {
+RMNode node = mock(RMNode.class);
+when(node.getTotalCapability()).thenReturn(Resource.newInstance(8192,
8));
+when(node.getHostName()).thenReturn("host.domain.com");
+return node;
+ }
+
+ private RMContainer createDefaultContainer() {
+return createContainer(Resource.newInstance(1024, 1), null);
+ }
+
+ private RMContainer createContainer(
+ Resource request, ApplicationAttemptId appAttemptId) {
+RMContainer container = mock(RMContainer.class);
+Container containerInner = mock(Container.class);
+ContainerId id = mock(ContainerId.class);
+when(id.getContainerId()).thenReturn(containerNum);
+when(containerInner.getResource()).
+thenReturn(Resources.clone(request));
+when(containerInner.getId()).thenReturn(id);
+when(containerInner.getExecutionType()).
+thenReturn(ExecutionType.GUARANTEED);
+when(container.getApplicationAttemptId()).thenReturn(appAttemptId);
+when(container.getContainerId()).thenReturn(id);
+when(container.getContainer()).thenReturn(containerInner);
+
when(container.getExecutionType()).thenReturn(ExecutionType.GUARANTEED);
+when(container.getAllocatedResource()).
+thenReturn(Resources.clone(request));
+containers.add(container);
+containerNum++;
+return container;
+ }
+
+ private void saturateCluster(FSSchedulerNode schedulerNode) {
+while (!Resources.isNone(schedulerNode.getUnallocatedResource())) {
+ createDefaultContainer();
+ schedulerNode.allocateContainer(containers.get((int)containerNum -
1));
+ schedulerNode.containerStarted(containers.get((int)containerNum - 1).
+ getContainerId());
+}
+ }
+
+ private FSAppAttempt createStarvingApp(FSSchedulerNode schedulerNode,
+ Resource request) {
+FSAppAttempt starvingApp = mock(FSAppAttempt.class);
+ApplicationAttemptId appAttemptId =
+mock(ApplicationAttemptId.class);
+when(starvingApp.getApplicationAttemptId()).thenReturn(appAttemptId);
+when(starvingApp.assignContainer(schedulerNode)).thenAnswer(
+new Answer() {
+ @Override
+ public Resource answer(InvocationOnMock invocationOnMock)
+ throws Throwable {
+Resource response = Resource.newInstance(0, 0);
+while (!Resources.isNone(request) &&
+!Resources.isNone(schedulerNode.getUnallocatedResource()))
{
+ RMContainer container = createContainer(request,
appAttemptId);
+ schedulerNode.allocateContainer(container);
+ Resources.addTo(response, container.getAllocatedResource());
+ Resources.subtractFrom(request,
+ container.getAllocatedResource());
+}
+return response;
+ }
+});
+when(starvingApp.getPendingDemand()).thenReturn(request);
+return starvingApp;
+ }
+
+ private void finalValidation(FSSchedulerNode schedulerNode) {
+assertEquals("Everything should
[jira] [Commented] (YARN-5829) FS preemption should reserve a node before considering containers on it for preemption
[
https://issues.apache.org/jira/browse/YARN-5829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15941499#comment-15941499
]
ASF GitHub Bot commented on YARN-5829:
--
Github user szegedim commented on a diff in the pull request:
https://github.com/apache/hadoop/pull/201#discussion_r108025258
--- Diff:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFSSchedulerNode.java
---
@@ -0,0 +1,376 @@
+package org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair;
+
+import org.apache.hadoop.yarn.api.records.*;
+import
org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainer;
+import org.apache.hadoop.yarn.server.resourcemanager.rmnode.RMNode;
+import org.apache.hadoop.yarn.util.resource.Resources;
+import org.junit.Test;
+import org.mockito.invocation.InvocationOnMock;
+import org.mockito.stubbing.Answer;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Map;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNotEquals;
+import static org.junit.Assert.assertTrue;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+/**
+ * Test scheduler node, especially preemption reservations.
+ */
+public class TestFSSchedulerNode {
+ private long containerNum = 0;
+ private ArrayList containers = new ArrayList<>();
+
+ private RMNode createNode() {
+RMNode node = mock(RMNode.class);
+when(node.getTotalCapability()).thenReturn(Resource.newInstance(8192,
8));
+when(node.getHostName()).thenReturn("host.domain.com");
+return node;
+ }
+
+ private RMContainer createDefaultContainer() {
+return createContainer(Resource.newInstance(1024, 1), null);
+ }
+
+ private RMContainer createContainer(
+ Resource request, ApplicationAttemptId appAttemptId) {
+RMContainer container = mock(RMContainer.class);
+Container containerInner = mock(Container.class);
+ContainerId id = mock(ContainerId.class);
+when(id.getContainerId()).thenReturn(containerNum);
+when(containerInner.getResource()).
+thenReturn(Resources.clone(request));
+when(containerInner.getId()).thenReturn(id);
+when(containerInner.getExecutionType()).
+thenReturn(ExecutionType.GUARANTEED);
+when(container.getApplicationAttemptId()).thenReturn(appAttemptId);
+when(container.getContainerId()).thenReturn(id);
+when(container.getContainer()).thenReturn(containerInner);
+
when(container.getExecutionType()).thenReturn(ExecutionType.GUARANTEED);
+when(container.getAllocatedResource()).
+thenReturn(Resources.clone(request));
+containers.add(container);
+containerNum++;
+return container;
+ }
+
+ private void saturateCluster(FSSchedulerNode schedulerNode) {
+while (!Resources.isNone(schedulerNode.getUnallocatedResource())) {
+ createDefaultContainer();
+ schedulerNode.allocateContainer(containers.get((int)containerNum -
1));
--- End diff --
I changed to int.
> FS preemption should reserve a node before considering containers on it for
> preemption
> --
>
> Key: YARN-5829
> URL: https://issues.apache.org/jira/browse/YARN-5829
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: fairscheduler
>Reporter: Karthik Kambatla
>Assignee: Miklos Szegedi
>
> FS preemption evaluates nodes for preemption, and subsequently preempts
> identified containers. If this node is not reserved for a specific
> application, any other application could be allocated resources on this node.
> Reserving the node for the starved application before preempting containers
> would help avoid this.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5829) FS preemption should reserve a node before considering containers on it for preemption
[
https://issues.apache.org/jira/browse/YARN-5829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15941497#comment-15941497
]
ASF GitHub Bot commented on YARN-5829:
--
Github user szegedim commented on a diff in the pull request:
https://github.com/apache/hadoop/pull/201#discussion_r108025247
--- Diff:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFSSchedulerNode.java
---
@@ -0,0 +1,376 @@
+package org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair;
+
+import org.apache.hadoop.yarn.api.records.*;
+import
org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainer;
+import org.apache.hadoop.yarn.server.resourcemanager.rmnode.RMNode;
+import org.apache.hadoop.yarn.util.resource.Resources;
+import org.junit.Test;
+import org.mockito.invocation.InvocationOnMock;
+import org.mockito.stubbing.Answer;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Map;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNotEquals;
+import static org.junit.Assert.assertTrue;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+/**
+ * Test scheduler node, especially preemption reservations.
+ */
+public class TestFSSchedulerNode {
+ private long containerNum = 0;
+ private ArrayList containers = new ArrayList<>();
+
+ private RMNode createNode() {
+RMNode node = mock(RMNode.class);
+when(node.getTotalCapability()).thenReturn(Resource.newInstance(8192,
8));
+when(node.getHostName()).thenReturn("host.domain.com");
+return node;
+ }
+
+ private RMContainer createDefaultContainer() {
+return createContainer(Resource.newInstance(1024, 1), null);
+ }
+
+ private RMContainer createContainer(
+ Resource request, ApplicationAttemptId appAttemptId) {
+RMContainer container = mock(RMContainer.class);
+Container containerInner = mock(Container.class);
+ContainerId id = mock(ContainerId.class);
+when(id.getContainerId()).thenReturn(containerNum);
+when(containerInner.getResource()).
+thenReturn(Resources.clone(request));
+when(containerInner.getId()).thenReturn(id);
+when(containerInner.getExecutionType()).
+thenReturn(ExecutionType.GUARANTEED);
+when(container.getApplicationAttemptId()).thenReturn(appAttemptId);
+when(container.getContainerId()).thenReturn(id);
+when(container.getContainer()).thenReturn(containerInner);
+
when(container.getExecutionType()).thenReturn(ExecutionType.GUARANTEED);
--- End diff --
That is for the inner class. Sorry, what do you mean? When?
> FS preemption should reserve a node before considering containers on it for
> preemption
> --
>
> Key: YARN-5829
> URL: https://issues.apache.org/jira/browse/YARN-5829
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: fairscheduler
>Reporter: Karthik Kambatla
>Assignee: Miklos Szegedi
>
> FS preemption evaluates nodes for preemption, and subsequently preempts
> identified containers. If this node is not reserved for a specific
> application, any other application could be allocated resources on this node.
> Reserving the node for the starved application before preempting containers
> would help avoid this.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5829) FS preemption should reserve a node before considering containers on it for preemption
[
https://issues.apache.org/jira/browse/YARN-5829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15941498#comment-15941498
]
ASF GitHub Bot commented on YARN-5829:
--
Github user szegedim commented on a diff in the pull request:
https://github.com/apache/hadoop/pull/201#discussion_r108025251
--- Diff:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFSSchedulerNode.java
---
@@ -0,0 +1,376 @@
+package org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair;
+
+import org.apache.hadoop.yarn.api.records.*;
+import
org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainer;
+import org.apache.hadoop.yarn.server.resourcemanager.rmnode.RMNode;
+import org.apache.hadoop.yarn.util.resource.Resources;
+import org.junit.Test;
+import org.mockito.invocation.InvocationOnMock;
+import org.mockito.stubbing.Answer;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Map;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNotEquals;
+import static org.junit.Assert.assertTrue;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+/**
+ * Test scheduler node, especially preemption reservations.
+ */
+public class TestFSSchedulerNode {
+ private long containerNum = 0;
+ private ArrayList containers = new ArrayList<>();
+
+ private RMNode createNode() {
+RMNode node = mock(RMNode.class);
+when(node.getTotalCapability()).thenReturn(Resource.newInstance(8192,
8));
+when(node.getHostName()).thenReturn("host.domain.com");
+return node;
+ }
+
+ private RMContainer createDefaultContainer() {
+return createContainer(Resource.newInstance(1024, 1), null);
+ }
+
+ private RMContainer createContainer(
+ Resource request, ApplicationAttemptId appAttemptId) {
+RMContainer container = mock(RMContainer.class);
+Container containerInner = mock(Container.class);
+ContainerId id = mock(ContainerId.class);
+when(id.getContainerId()).thenReturn(containerNum);
+when(containerInner.getResource()).
+thenReturn(Resources.clone(request));
+when(containerInner.getId()).thenReturn(id);
+when(containerInner.getExecutionType()).
+thenReturn(ExecutionType.GUARANTEED);
+when(container.getApplicationAttemptId()).thenReturn(appAttemptId);
+when(container.getContainerId()).thenReturn(id);
+when(container.getContainer()).thenReturn(containerInner);
+
when(container.getExecutionType()).thenReturn(ExecutionType.GUARANTEED);
--- End diff --
I do have a when in all these cases.
> FS preemption should reserve a node before considering containers on it for
> preemption
> --
>
> Key: YARN-5829
> URL: https://issues.apache.org/jira/browse/YARN-5829
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: fairscheduler
>Reporter: Karthik Kambatla
>Assignee: Miklos Szegedi
>
> FS preemption evaluates nodes for preemption, and subsequently preempts
> identified containers. If this node is not reserved for a specific
> application, any other application could be allocated resources on this node.
> Reserving the node for the starved application before preempting containers
> would help avoid this.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5829) FS preemption should reserve a node before considering containers on it for preemption
[
https://issues.apache.org/jira/browse/YARN-5829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15941495#comment-15941495
]
ASF GitHub Bot commented on YARN-5829:
--
Github user szegedim commented on a diff in the pull request:
https://github.com/apache/hadoop/pull/201#discussion_r108025201
--- Diff:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFSSchedulerNode.java
---
@@ -0,0 +1,376 @@
+package org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair;
+
+import org.apache.hadoop.yarn.api.records.*;
+import
org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainer;
+import org.apache.hadoop.yarn.server.resourcemanager.rmnode.RMNode;
+import org.apache.hadoop.yarn.util.resource.Resources;
+import org.junit.Test;
+import org.mockito.invocation.InvocationOnMock;
+import org.mockito.stubbing.Answer;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Map;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNotEquals;
+import static org.junit.Assert.assertTrue;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+/**
+ * Test scheduler node, especially preemption reservations.
+ */
+public class TestFSSchedulerNode {
+ private long containerNum = 0;
+ private ArrayList containers = new ArrayList<>();
--- End diff --
Done.
> FS preemption should reserve a node before considering containers on it for
> preemption
> --
>
> Key: YARN-5829
> URL: https://issues.apache.org/jira/browse/YARN-5829
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: fairscheduler
>Reporter: Karthik Kambatla
>Assignee: Miklos Szegedi
>
> FS preemption evaluates nodes for preemption, and subsequently preempts
> identified containers. If this node is not reserved for a specific
> application, any other application could be allocated resources on this node.
> Reserving the node for the starved application before preempting containers
> would help avoid this.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5829) FS preemption should reserve a node before considering containers on it for preemption
[
https://issues.apache.org/jira/browse/YARN-5829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15941491#comment-15941491
]
ASF GitHub Bot commented on YARN-5829:
--
Github user szegedim commented on a diff in the pull request:
https://github.com/apache/hadoop/pull/201#discussion_r108025184
--- Diff:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFSSchedulerNode.java
---
@@ -0,0 +1,376 @@
+package org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair;
+
+import org.apache.hadoop.yarn.api.records.*;
+import
org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainer;
+import org.apache.hadoop.yarn.server.resourcemanager.rmnode.RMNode;
+import org.apache.hadoop.yarn.util.resource.Resources;
+import org.junit.Test;
+import org.mockito.invocation.InvocationOnMock;
+import org.mockito.stubbing.Answer;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Map;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNotEquals;
+import static org.junit.Assert.assertTrue;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+/**
+ * Test scheduler node, especially preemption reservations.
+ */
+public class TestFSSchedulerNode {
+ private long containerNum = 0;
--- End diff --
Done.
> FS preemption should reserve a node before considering containers on it for
> preemption
> --
>
> Key: YARN-5829
> URL: https://issues.apache.org/jira/browse/YARN-5829
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: fairscheduler
>Reporter: Karthik Kambatla
>Assignee: Miklos Szegedi
>
> FS preemption evaluates nodes for preemption, and subsequently preempts
> identified containers. If this node is not reserved for a specific
> application, any other application could be allocated resources on this node.
> Reserving the node for the starved application before preempting containers
> would help avoid this.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5685) Non-embedded HA failover is broken
[
https://issues.apache.org/jira/browse/YARN-5685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15941490#comment-15941490
]
Hadoop QA commented on YARN-5685:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
13s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
0s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 12m
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
48s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
6s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
30s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
7s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 8m 7s{color}
| {color:red} hadoop-yarn-project_hadoop-yarn generated 10 new + 35 unchanged -
0 fixed = 45 total (was 35) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
52s{color} | {color:green} hadoop-yarn-project/hadoop-yarn: The patch generated
0 new + 260 unchanged - 4 fixed = 260 total (was 264) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
33s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
36s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 41m 27s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
33s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}105m 2s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.yarn.server.resourcemanager.TestRMRestart |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | YARN-5685 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12860453/YARN-5685.005.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux bf5c134f215a 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12
13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 1f66524 |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| javac |
https://builds.apache.org/job/PreCommit-YARN-Build/15385/artifact/patchprocess/diff-compile-javac-hadoop-yarn-project_hadoop-yarn.txt
|
| unit |
[jira] [Commented] (YARN-5829) FS preemption should reserve a node before considering containers on it for preemption
[
https://issues.apache.org/jira/browse/YARN-5829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15941481#comment-15941481
]
ASF GitHub Bot commented on YARN-5829:
--
Github user szegedim commented on a diff in the pull request:
https://github.com/apache/hadoop/pull/201#discussion_r108024837
--- Diff:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairScheduler.java
---
@@ -968,6 +966,32 @@ private boolean shouldContinueAssigning(int containers,
}
}
+ /**
+ * Assign preempted containers to the applications that have reserved
+ * resources for preempted containers.
+ * @param node Node to check
+ * @return assignment has occurred
+ */
+ static boolean assignPreemptedContainers(FSSchedulerNode node ) {
+boolean assignedAny = false;
+node.cleanupPreemptionList();
+for (Entry entry :
+node.getPreemptionList().entrySet()) {
+ FSAppAttempt app = entry.getKey();
+ Resource reserved = Resources.clone(entry.getValue());
+ while (!app.isStopped() && !Resources.isNone(reserved)) {
+Resource assigned = app.assignContainer(node);
+if (Resources.isNone(assigned)) {
+ break;
+}
+assignedAny = true;
+Resources.subtractFromNonNegative(reserved, assigned);
+ }
+}
+node.cleanupPreemptionList();
--- End diff --
Done.
> FS preemption should reserve a node before considering containers on it for
> preemption
> --
>
> Key: YARN-5829
> URL: https://issues.apache.org/jira/browse/YARN-5829
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: fairscheduler
>Reporter: Karthik Kambatla
>Assignee: Miklos Szegedi
>
> FS preemption evaluates nodes for preemption, and subsequently preempts
> identified containers. If this node is not reserved for a specific
> application, any other application could be allocated resources on this node.
> Reserving the node for the starved application before preempting containers
> would help avoid this.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5829) FS preemption should reserve a node before considering containers on it for preemption
[
https://issues.apache.org/jira/browse/YARN-5829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15941483#comment-15941483
]
ASF GitHub Bot commented on YARN-5829:
--
Github user szegedim commented on a diff in the pull request:
https://github.com/apache/hadoop/pull/201#discussion_r108024844
--- Diff:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSAppAttempt.java
---
@@ -646,7 +646,6 @@ private Container createContainer(FSSchedulerNode node,
Resource capability,
private boolean reserve(Resource perAllocationResource, FSSchedulerNode
node,
Container reservedContainer, NodeType type,
SchedulerRequestKey schedulerKey) {
-
--- End diff --
Done.
> FS preemption should reserve a node before considering containers on it for
> preemption
> --
>
> Key: YARN-5829
> URL: https://issues.apache.org/jira/browse/YARN-5829
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: fairscheduler
>Reporter: Karthik Kambatla
>Assignee: Miklos Szegedi
>
> FS preemption evaluates nodes for preemption, and subsequently preempts
> identified containers. If this node is not reserved for a specific
> application, any other application could be allocated resources on this node.
> Reserving the node for the starved application before preempting containers
> would help avoid this.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5829) FS preemption should reserve a node before considering containers on it for preemption
[
https://issues.apache.org/jira/browse/YARN-5829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15941480#comment-15941480
]
ASF GitHub Bot commented on YARN-5829:
--
Github user szegedim commented on a diff in the pull request:
https://github.com/apache/hadoop/pull/201#discussion_r108024823
--- Diff:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairScheduler.java
---
@@ -968,6 +966,32 @@ private boolean shouldContinueAssigning(int containers,
}
}
+ /**
+ * Assign preempted containers to the applications that have reserved
+ * resources for preempted containers.
+ * @param node Node to check
+ * @return assignment has occurred
+ */
+ static boolean assignPreemptedContainers(FSSchedulerNode node ) {
+boolean assignedAny = false;
+node.cleanupPreemptionList();
+for (Entry entry :
+node.getPreemptionList().entrySet()) {
+ FSAppAttempt app = entry.getKey();
+ Resource reserved = Resources.clone(entry.getValue());
+ while (!app.isStopped() && !Resources.isNone(reserved)) {
+Resource assigned = app.assignContainer(node);
+if (Resources.isNone(assigned)) {
+ break;
--- End diff --
Done.
> FS preemption should reserve a node before considering containers on it for
> preemption
> --
>
> Key: YARN-5829
> URL: https://issues.apache.org/jira/browse/YARN-5829
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: fairscheduler
>Reporter: Karthik Kambatla
>Assignee: Miklos Szegedi
>
> FS preemption evaluates nodes for preemption, and subsequently preempts
> identified containers. If this node is not reserved for a specific
> application, any other application could be allocated resources on this node.
> Reserving the node for the starved application before preempting containers
> would help avoid this.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5829) FS preemption should reserve a node before considering containers on it for preemption
[
https://issues.apache.org/jira/browse/YARN-5829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15941479#comment-15941479
]
ASF GitHub Bot commented on YARN-5829:
--
Github user szegedim commented on a diff in the pull request:
https://github.com/apache/hadoop/pull/201#discussion_r108024815
--- Diff:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairScheduler.java
---
@@ -968,6 +966,32 @@ private boolean shouldContinueAssigning(int containers,
}
}
+ /**
+ * Assign preempted containers to the applications that have reserved
+ * resources for preempted containers.
+ * @param node Node to check
+ * @return assignment has occurred
+ */
+ static boolean assignPreemptedContainers(FSSchedulerNode node ) {
+boolean assignedAny = false;
+node.cleanupPreemptionList();
+for (Entry entry :
+node.getPreemptionList().entrySet()) {
+ FSAppAttempt app = entry.getKey();
+ Resource reserved = Resources.clone(entry.getValue());
--- End diff --
Done.
> FS preemption should reserve a node before considering containers on it for
> preemption
> --
>
> Key: YARN-5829
> URL: https://issues.apache.org/jira/browse/YARN-5829
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: fairscheduler
>Reporter: Karthik Kambatla
>Assignee: Miklos Szegedi
>
> FS preemption evaluates nodes for preemption, and subsequently preempts
> identified containers. If this node is not reserved for a specific
> application, any other application could be allocated resources on this node.
> Reserving the node for the starved application before preempting containers
> would help avoid this.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5829) FS preemption should reserve a node before considering containers on it for preemption
[
https://issues.apache.org/jira/browse/YARN-5829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15941477#comment-15941477
]
ASF GitHub Bot commented on YARN-5829:
--
Github user szegedim commented on a diff in the pull request:
https://github.com/apache/hadoop/pull/201#discussion_r108024812
--- Diff:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairScheduler.java
---
@@ -968,6 +966,32 @@ private boolean shouldContinueAssigning(int containers,
}
}
+ /**
+ * Assign preempted containers to the applications that have reserved
+ * resources for preempted containers.
+ * @param node Node to check
+ * @return assignment has occurred
+ */
+ static boolean assignPreemptedContainers(FSSchedulerNode node ) {
+boolean assignedAny = false;
+node.cleanupPreemptionList();
--- End diff --
Done.
> FS preemption should reserve a node before considering containers on it for
> preemption
> --
>
> Key: YARN-5829
> URL: https://issues.apache.org/jira/browse/YARN-5829
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: fairscheduler
>Reporter: Karthik Kambatla
>Assignee: Miklos Szegedi
>
> FS preemption evaluates nodes for preemption, and subsequently preempts
> identified containers. If this node is not reserved for a specific
> application, any other application could be allocated resources on this node.
> Reserving the node for the starved application before preempting containers
> would help avoid this.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5829) FS preemption should reserve a node before considering containers on it for preemption
[
https://issues.apache.org/jira/browse/YARN-5829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15941474#comment-15941474
]
ASF GitHub Bot commented on YARN-5829:
--
Github user szegedim commented on a diff in the pull request:
https://github.com/apache/hadoop/pull/201#discussion_r108024779
--- Diff:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSSchedulerNode.java
---
@@ -110,16 +149,57 @@ synchronized FSAppAttempt getReservedAppSchedulable()
{
}
/**
+ * List reserved resources after preemption and assign them to the
+ * appropriate applications in a FIFO order.
+ * It is a good practice to call {@link #cleanupPreemptionList()}
+ * after this call
+ * @return if any resources were allocated
+ */
+ synchronized LinkedHashMap getPreemptionList() {
+cleanupPreemptionList();
+return new LinkedHashMap<>(resourcesPreemptedForApp);
+ }
+
+ /**
+ * Remove apps that have their preemption requests fulfilled
+ */
+ synchronized void cleanupPreemptionList() {
+Iterator iterator =
+resourcesPreemptedForApp.keySet().iterator();
+while (iterator.hasNext()) {
+ FSAppAttempt app = iterator.next();
+ boolean removeApp = false;
+ if (app.isStopped() || Resources.equals(
+ app.getPendingDemand(), Resources.none())) {
+// App does not need more resources
+Resources.subtractFrom(totalResourcesPreempted,
+resourcesPreemptedForApp.get(app));
+iterator.remove();
+ }
+}
+ }
+
+ /**
* Mark {@code containers} as being considered for preemption so they are
* not considered again. A call to this requires a corresponding call to
- * {@link #removeContainerForPreemption} to ensure we do not mark a
+ * removeContainer for preemption to ensure we do not mark a
--- End diff --
Fixed.
> FS preemption should reserve a node before considering containers on it for
> preemption
> --
>
> Key: YARN-5829
> URL: https://issues.apache.org/jira/browse/YARN-5829
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: fairscheduler
>Reporter: Karthik Kambatla
>Assignee: Miklos Szegedi
>
> FS preemption evaluates nodes for preemption, and subsequently preempts
> identified containers. If this node is not reserved for a specific
> application, any other application could be allocated resources on this node.
> Reserving the node for the starved application before preempting containers
> would help avoid this.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5829) FS preemption should reserve a node before considering containers on it for preemption
[
https://issues.apache.org/jira/browse/YARN-5829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15941472#comment-15941472
]
ASF GitHub Bot commented on YARN-5829:
--
Github user szegedim commented on a diff in the pull request:
https://github.com/apache/hadoop/pull/201#discussion_r108024723
--- Diff:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSSchedulerNode.java
---
@@ -110,16 +149,57 @@ synchronized FSAppAttempt getReservedAppSchedulable()
{
}
/**
+ * List reserved resources after preemption and assign them to the
+ * appropriate applications in a FIFO order.
+ * It is a good practice to call {@link #cleanupPreemptionList()}
+ * after this call
+ * @return if any resources were allocated
+ */
+ synchronized LinkedHashMap getPreemptionList() {
+cleanupPreemptionList();
+return new LinkedHashMap<>(resourcesPreemptedForApp);
+ }
+
+ /**
+ * Remove apps that have their preemption requests fulfilled
+ */
+ synchronized void cleanupPreemptionList() {
+Iterator iterator =
+resourcesPreemptedForApp.keySet().iterator();
+while (iterator.hasNext()) {
+ FSAppAttempt app = iterator.next();
+ boolean removeApp = false;
+ if (app.isStopped() || Resources.equals(
--- End diff --
Done.
> FS preemption should reserve a node before considering containers on it for
> preemption
> --
>
> Key: YARN-5829
> URL: https://issues.apache.org/jira/browse/YARN-5829
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: fairscheduler
>Reporter: Karthik Kambatla
>Assignee: Miklos Szegedi
>
> FS preemption evaluates nodes for preemption, and subsequently preempts
> identified containers. If this node is not reserved for a specific
> application, any other application could be allocated resources on this node.
> Reserving the node for the starved application before preempting containers
> would help avoid this.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5829) FS preemption should reserve a node before considering containers on it for preemption
[
https://issues.apache.org/jira/browse/YARN-5829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15941470#comment-15941470
]
ASF GitHub Bot commented on YARN-5829:
--
Github user szegedim commented on a diff in the pull request:
https://github.com/apache/hadoop/pull/201#discussion_r108024707
--- Diff:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSSchedulerNode.java
---
@@ -110,16 +149,57 @@ synchronized FSAppAttempt getReservedAppSchedulable()
{
}
/**
+ * List reserved resources after preemption and assign them to the
+ * appropriate applications in a FIFO order.
+ * It is a good practice to call {@link #cleanupPreemptionList()}
+ * after this call
+ * @return if any resources were allocated
+ */
+ synchronized LinkedHashMap getPreemptionList() {
+cleanupPreemptionList();
+return new LinkedHashMap<>(resourcesPreemptedForApp);
+ }
+
+ /**
+ * Remove apps that have their preemption requests fulfilled
+ */
+ synchronized void cleanupPreemptionList() {
+Iterator iterator =
+resourcesPreemptedForApp.keySet().iterator();
+while (iterator.hasNext()) {
+ FSAppAttempt app = iterator.next();
+ boolean removeApp = false;
--- End diff --
Done.
> FS preemption should reserve a node before considering containers on it for
> preemption
> --
>
> Key: YARN-5829
> URL: https://issues.apache.org/jira/browse/YARN-5829
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: fairscheduler
>Reporter: Karthik Kambatla
>Assignee: Miklos Szegedi
>
> FS preemption evaluates nodes for preemption, and subsequently preempts
> identified containers. If this node is not reserved for a specific
> application, any other application could be allocated resources on this node.
> Reserving the node for the starved application before preempting containers
> would help avoid this.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5829) FS preemption should reserve a node before considering containers on it for preemption
[
https://issues.apache.org/jira/browse/YARN-5829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15941469#comment-15941469
]
ASF GitHub Bot commented on YARN-5829:
--
Github user szegedim commented on a diff in the pull request:
https://github.com/apache/hadoop/pull/201#discussion_r108024703
--- Diff:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSSchedulerNode.java
---
@@ -110,16 +149,57 @@ synchronized FSAppAttempt getReservedAppSchedulable()
{
}
/**
+ * List reserved resources after preemption and assign them to the
+ * appropriate applications in a FIFO order.
+ * It is a good practice to call {@link #cleanupPreemptionList()}
+ * after this call
+ * @return if any resources were allocated
+ */
+ synchronized LinkedHashMap getPreemptionList() {
+cleanupPreemptionList();
+return new LinkedHashMap<>(resourcesPreemptedForApp);
+ }
+
+ /**
+ * Remove apps that have their preemption requests fulfilled
+ */
+ synchronized void cleanupPreemptionList() {
--- End diff --
Done.
> FS preemption should reserve a node before considering containers on it for
> preemption
> --
>
> Key: YARN-5829
> URL: https://issues.apache.org/jira/browse/YARN-5829
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: fairscheduler
>Reporter: Karthik Kambatla
>Assignee: Miklos Szegedi
>
> FS preemption evaluates nodes for preemption, and subsequently preempts
> identified containers. If this node is not reserved for a specific
> application, any other application could be allocated resources on this node.
> Reserving the node for the starved application before preempting containers
> would help avoid this.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5829) FS preemption should reserve a node before considering containers on it for preemption
[
https://issues.apache.org/jira/browse/YARN-5829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15941468#comment-15941468
]
ASF GitHub Bot commented on YARN-5829:
--
Github user szegedim commented on a diff in the pull request:
https://github.com/apache/hadoop/pull/201#discussion_r108024698
--- Diff:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSSchedulerNode.java
---
@@ -110,16 +149,57 @@ synchronized FSAppAttempt getReservedAppSchedulable()
{
}
/**
+ * List reserved resources after preemption and assign them to the
+ * appropriate applications in a FIFO order.
+ * It is a good practice to call {@link #cleanupPreemptionList()}
--- End diff --
Fixed.
> FS preemption should reserve a node before considering containers on it for
> preemption
> --
>
> Key: YARN-5829
> URL: https://issues.apache.org/jira/browse/YARN-5829
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: fairscheduler
>Reporter: Karthik Kambatla
>Assignee: Miklos Szegedi
>
> FS preemption evaluates nodes for preemption, and subsequently preempts
> identified containers. If this node is not reserved for a specific
> application, any other application could be allocated resources on this node.
> Reserving the node for the starved application before preempting containers
> would help avoid this.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5829) FS preemption should reserve a node before considering containers on it for preemption
[
https://issues.apache.org/jira/browse/YARN-5829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15941467#comment-15941467
]
ASF GitHub Bot commented on YARN-5829:
--
Github user szegedim commented on a diff in the pull request:
https://github.com/apache/hadoop/pull/201#discussion_r108024673
--- Diff:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSSchedulerNode.java
---
@@ -38,15 +47,45 @@
public class FSSchedulerNode extends SchedulerNode {
private static final Log LOG = LogFactory.getLog(FSSchedulerNode.class);
+ private static final Comparator comparator =
+ new Comparator() {
+@Override
+public int compare(RMContainer o1, RMContainer o2) {
+ return Long.compare(o1.getContainerId().getContainerId(),
+ o2.getContainerId().getContainerId());
+}
+ };
private FSAppAttempt reservedAppSchedulable;
- private final Set containersForPreemption =
- new ConcurrentSkipListSet<>();
+ // Stores list of containers still to be preempted
+ @VisibleForTesting
+ final Set containersForPreemption =
+ new ConcurrentSkipListSet<>(comparator);
+ // Stores amount of respources preempted and reserved for each app
+ @VisibleForTesting
+ final Map
+ resourcesPreemptedForApp = new LinkedHashMap<>();
+ // Sum of resourcesPreemptedForApp values, total resources that are
+ // slated for preemption
+ Resource totalResourcesPreempted = Resource.newInstance(0, 0);
public FSSchedulerNode(RMNode node, boolean usePortForNodeName) {
super(node, usePortForNodeName);
}
+ /**
+ * Total amount of reserved resources including reservations and
preempted
+ * containers.
+ * @return total resources reserved
+ */
+ Resource getTotalReserved() {
+Resource totalReserved = getReservedContainer() != null
+? Resources.clone(getReservedContainer().getAllocatedResource())
+: Resource.newInstance(0, 0);
--- End diff --
Fixed.
> FS preemption should reserve a node before considering containers on it for
> preemption
> --
>
> Key: YARN-5829
> URL: https://issues.apache.org/jira/browse/YARN-5829
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: fairscheduler
>Reporter: Karthik Kambatla
>Assignee: Miklos Szegedi
>
> FS preemption evaluates nodes for preemption, and subsequently preempts
> identified containers. If this node is not reserved for a specific
> application, any other application could be allocated resources on this node.
> Reserving the node for the starved application before preempting containers
> would help avoid this.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5829) FS preemption should reserve a node before considering containers on it for preemption
[
https://issues.apache.org/jira/browse/YARN-5829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15941465#comment-15941465
]
ASF GitHub Bot commented on YARN-5829:
--
Github user szegedim commented on a diff in the pull request:
https://github.com/apache/hadoop/pull/201#discussion_r108024599
--- Diff:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSSchedulerNode.java
---
@@ -38,15 +47,45 @@
public class FSSchedulerNode extends SchedulerNode {
private static final Log LOG = LogFactory.getLog(FSSchedulerNode.class);
+ private static final Comparator comparator =
+ new Comparator() {
+@Override
+public int compare(RMContainer o1, RMContainer o2) {
+ return Long.compare(o1.getContainerId().getContainerId(),
+ o2.getContainerId().getContainerId());
+}
+ };
private FSAppAttempt reservedAppSchedulable;
- private final Set containersForPreemption =
- new ConcurrentSkipListSet<>();
+ // Stores list of containers still to be preempted
+ @VisibleForTesting
+ final Set containersForPreemption =
+ new ConcurrentSkipListSet<>(comparator);
+ // Stores amount of respources preempted and reserved for each app
--- End diff --
Fixed.
> FS preemption should reserve a node before considering containers on it for
> preemption
> --
>
> Key: YARN-5829
> URL: https://issues.apache.org/jira/browse/YARN-5829
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: fairscheduler
>Reporter: Karthik Kambatla
>Assignee: Miklos Szegedi
>
> FS preemption evaluates nodes for preemption, and subsequently preempts
> identified containers. If this node is not reserved for a specific
> application, any other application could be allocated resources on this node.
> Reserving the node for the starved application before preempting containers
> would help avoid this.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5829) FS preemption should reserve a node before considering containers on it for preemption
[
https://issues.apache.org/jira/browse/YARN-5829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15941450#comment-15941450
]
ASF GitHub Bot commented on YARN-5829:
--
Github user szegedim commented on a diff in the pull request:
https://github.com/apache/hadoop/pull/201#discussion_r108024135
--- Diff:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSSchedulerNode.java
---
@@ -38,15 +47,45 @@
public class FSSchedulerNode extends SchedulerNode {
private static final Log LOG = LogFactory.getLog(FSSchedulerNode.class);
+ private static final Comparator comparator =
--- End diff --
Done.
> FS preemption should reserve a node before considering containers on it for
> preemption
> --
>
> Key: YARN-5829
> URL: https://issues.apache.org/jira/browse/YARN-5829
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: fairscheduler
>Reporter: Karthik Kambatla
>Assignee: Miklos Szegedi
>
> FS preemption evaluates nodes for preemption, and subsequently preempts
> identified containers. If this node is not reserved for a specific
> application, any other application could be allocated resources on this node.
> Reserving the node for the starved application before preempting containers
> would help avoid this.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5829) FS preemption should reserve a node before considering containers on it for preemption
[ https://issues.apache.org/jira/browse/YARN-5829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15941451#comment-15941451 ] ASF GitHub Bot commented on YARN-5829: -- Github user szegedim commented on a diff in the pull request: https://github.com/apache/hadoop/pull/201#discussion_r108024138 --- Diff: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSSchedulerNode.java --- @@ -18,18 +18,27 @@ package org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair; +import com.google.common.annotations.VisibleForTesting; import org.apache.commons.logging.Log; import org.apache.commons.logging.LogFactory; import org.apache.hadoop.classification.InterfaceAudience.Private; import org.apache.hadoop.classification.InterfaceStability.Unstable; import org.apache.hadoop.yarn.api.records.ApplicationAttemptId; +import org.apache.hadoop.yarn.api.records.ContainerId; +import org.apache.hadoop.yarn.api.records.Resource; import org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainer; import org.apache.hadoop.yarn.server.resourcemanager.rmnode.RMNode; import org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerApplicationAttempt; import org.apache.hadoop.yarn.server.scheduler.SchedulerRequestKey; import org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerNode; +import org.apache.hadoop.yarn.util.resource.Resources; import java.util.Collection; +import java.util.Comparator; +import java.util.Iterator; +import java.util.LinkedHashMap; +import java.util.LinkedHashSet; --- End diff -- Done. > FS preemption should reserve a node before considering containers on it for > preemption > -- > > Key: YARN-5829 > URL: https://issues.apache.org/jira/browse/YARN-5829 > Project: Hadoop YARN > Issue Type: Sub-task > Components: fairscheduler >Reporter: Karthik Kambatla >Assignee: Miklos Szegedi > > FS preemption evaluates nodes for preemption, and subsequently preempts > identified containers. If this node is not reserved for a specific > application, any other application could be allocated resources on this node. > Reserving the node for the starved application before preempting containers > would help avoid this. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5829) FS preemption should reserve a node before considering containers on it for preemption
[
https://issues.apache.org/jira/browse/YARN-5829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15941447#comment-15941447
]
ASF GitHub Bot commented on YARN-5829:
--
Github user szegedim commented on a diff in the pull request:
https://github.com/apache/hadoop/pull/201#discussion_r108024058
--- Diff:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSSchedulerNode.java
---
@@ -131,10 +203,58 @@ void
addContainersForPreemption(Collection containers) {
/**
* Remove container from the set of containers marked for preemption.
+ * Reserve the preempted resources for the app that requested
+ * the preemption.
*
* @param container container to remove
*/
void removeContainerForPreemption(RMContainer container) {
containersForPreemption.remove(container);
}
+
+ /**
+ * The Scheduler has allocated containers on this node to the given
+ * application.
+ * @param rmContainer Allocated container
+ * @param launchedOnNode True if the container has been launched
+ */
+ @Override
+ protected synchronized void allocateContainer(RMContainer rmContainer,
+boolean launchedOnNode) {
+super.allocateContainer(rmContainer, launchedOnNode);
+Resource allocated = rmContainer.getAllocatedResource();
+if (!Resources.isNone(allocated)) {
+ for (FSAppAttempt app: resourcesPreemptedPerApp.keySet()) {
--- End diff --
All right. Fixed.
> FS preemption should reserve a node before considering containers on it for
> preemption
> --
>
> Key: YARN-5829
> URL: https://issues.apache.org/jira/browse/YARN-5829
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: fairscheduler
>Reporter: Karthik Kambatla
>Assignee: Miklos Szegedi
>
> FS preemption evaluates nodes for preemption, and subsequently preempts
> identified containers. If this node is not reserved for a specific
> application, any other application could be allocated resources on this node.
> Reserving the node for the starved application before preempting containers
> would help avoid this.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5829) FS preemption should reserve a node before considering containers on it for preemption
[
https://issues.apache.org/jira/browse/YARN-5829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15941421#comment-15941421
]
ASF GitHub Bot commented on YARN-5829:
--
Github user kambatla commented on a diff in the pull request:
https://github.com/apache/hadoop/pull/201#discussion_r108021362
--- Diff:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairScheduler.java
---
@@ -968,6 +966,32 @@ private boolean shouldContinueAssigning(int containers,
}
}
+ /**
+ * Assign preempted containers to the applications that have reserved
+ * resources for preempted containers.
+ * @param node Node to check
+ * @return assignment has occurred
+ */
+ static boolean assignPreemptedContainers(FSSchedulerNode node ) {
+boolean assignedAny = false;
+node.cleanupPreemptionList();
+for (Entry entry :
+node.getPreemptionList().entrySet()) {
+ FSAppAttempt app = entry.getKey();
+ Resource reserved = Resources.clone(entry.getValue());
+ while (!app.isStopped() && !Resources.isNone(reserved)) {
+Resource assigned = app.assignContainer(node);
+if (Resources.isNone(assigned)) {
+ break;
+}
+assignedAny = true;
+Resources.subtractFromNonNegative(reserved, assigned);
+ }
+}
+node.cleanupPreemptionList();
--- End diff --
Again, let us avoid cleaning up explicitly.
> FS preemption should reserve a node before considering containers on it for
> preemption
> --
>
> Key: YARN-5829
> URL: https://issues.apache.org/jira/browse/YARN-5829
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: fairscheduler
>Reporter: Karthik Kambatla
>Assignee: Miklos Szegedi
>
> FS preemption evaluates nodes for preemption, and subsequently preempts
> identified containers. If this node is not reserved for a specific
> application, any other application could be allocated resources on this node.
> Reserving the node for the starved application before preempting containers
> would help avoid this.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5829) FS preemption should reserve a node before considering containers on it for preemption
[
https://issues.apache.org/jira/browse/YARN-5829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15941435#comment-15941435
]
ASF GitHub Bot commented on YARN-5829:
--
Github user kambatla commented on a diff in the pull request:
https://github.com/apache/hadoop/pull/201#discussion_r108020752
--- Diff:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSSchedulerNode.java
---
@@ -110,16 +149,57 @@ synchronized FSAppAttempt getReservedAppSchedulable()
{
}
/**
+ * List reserved resources after preemption and assign them to the
+ * appropriate applications in a FIFO order.
+ * It is a good practice to call {@link #cleanupPreemptionList()}
--- End diff --
Having to call cleanup before get seems like a bug-magnet.
Since get already calls cleanup, I recommend not suggesting that callers
call cleanup.
> FS preemption should reserve a node before considering containers on it for
> preemption
> --
>
> Key: YARN-5829
> URL: https://issues.apache.org/jira/browse/YARN-5829
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: fairscheduler
>Reporter: Karthik Kambatla
>Assignee: Miklos Szegedi
>
> FS preemption evaluates nodes for preemption, and subsequently preempts
> identified containers. If this node is not reserved for a specific
> application, any other application could be allocated resources on this node.
> Reserving the node for the starved application before preempting containers
> would help avoid this.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5829) FS preemption should reserve a node before considering containers on it for preemption
[
https://issues.apache.org/jira/browse/YARN-5829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15941440#comment-15941440
]
ASF GitHub Bot commented on YARN-5829:
--
Github user kambatla commented on a diff in the pull request:
https://github.com/apache/hadoop/pull/201#discussion_r108023588
--- Diff:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFSSchedulerNode.java
---
@@ -0,0 +1,376 @@
+package org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair;
+
+import org.apache.hadoop.yarn.api.records.*;
+import
org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainer;
+import org.apache.hadoop.yarn.server.resourcemanager.rmnode.RMNode;
+import org.apache.hadoop.yarn.util.resource.Resources;
+import org.junit.Test;
+import org.mockito.invocation.InvocationOnMock;
+import org.mockito.stubbing.Answer;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Map;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNotEquals;
+import static org.junit.Assert.assertTrue;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+/**
+ * Test scheduler node, especially preemption reservations.
+ */
+public class TestFSSchedulerNode {
+ private long containerNum = 0;
+ private ArrayList containers = new ArrayList<>();
+
+ private RMNode createNode() {
+RMNode node = mock(RMNode.class);
+when(node.getTotalCapability()).thenReturn(Resource.newInstance(8192,
8));
+when(node.getHostName()).thenReturn("host.domain.com");
+return node;
+ }
+
+ private RMContainer createDefaultContainer() {
+return createContainer(Resource.newInstance(1024, 1), null);
+ }
+
+ private RMContainer createContainer(
+ Resource request, ApplicationAttemptId appAttemptId) {
+RMContainer container = mock(RMContainer.class);
+Container containerInner = mock(Container.class);
+ContainerId id = mock(ContainerId.class);
+when(id.getContainerId()).thenReturn(containerNum);
+when(containerInner.getResource()).
+thenReturn(Resources.clone(request));
+when(containerInner.getId()).thenReturn(id);
+when(containerInner.getExecutionType()).
+thenReturn(ExecutionType.GUARANTEED);
+when(container.getApplicationAttemptId()).thenReturn(appAttemptId);
+when(container.getContainerId()).thenReturn(id);
+when(container.getContainer()).thenReturn(containerInner);
+
when(container.getExecutionType()).thenReturn(ExecutionType.GUARANTEED);
+when(container.getAllocatedResource()).
+thenReturn(Resources.clone(request));
+containers.add(container);
+containerNum++;
+return container;
+ }
+
+ private void saturateCluster(FSSchedulerNode schedulerNode) {
+while (!Resources.isNone(schedulerNode.getUnallocatedResource())) {
+ createDefaultContainer();
+ schedulerNode.allocateContainer(containers.get((int)containerNum -
1));
+ schedulerNode.containerStarted(containers.get((int)containerNum - 1).
+ getContainerId());
+}
+ }
+
+ private FSAppAttempt createStarvingApp(FSSchedulerNode schedulerNode,
+ Resource request) {
+FSAppAttempt starvingApp = mock(FSAppAttempt.class);
+ApplicationAttemptId appAttemptId =
+mock(ApplicationAttemptId.class);
+when(starvingApp.getApplicationAttemptId()).thenReturn(appAttemptId);
+when(starvingApp.assignContainer(schedulerNode)).thenAnswer(
+new Answer() {
+ @Override
+ public Resource answer(InvocationOnMock invocationOnMock)
+ throws Throwable {
+Resource response = Resource.newInstance(0, 0);
+while (!Resources.isNone(request) &&
+!Resources.isNone(schedulerNode.getUnallocatedResource()))
{
+ RMContainer container = createContainer(request,
appAttemptId);
+ schedulerNode.allocateContainer(container);
+ Resources.addTo(response, container.getAllocatedResource());
+ Resources.subtractFrom(request,
+ container.getAllocatedResource());
+}
+return response;
+ }
+});
+when(starvingApp.getPendingDemand()).thenReturn(request);
+return starvingApp;
+ }
+
+ private void finalValidation(FSSchedulerNode schedulerNode) {
+assertEquals("Everything should
[jira] [Commented] (YARN-5829) FS preemption should reserve a node before considering containers on it for preemption
[
https://issues.apache.org/jira/browse/YARN-5829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15941441#comment-15941441
]
ASF GitHub Bot commented on YARN-5829:
--
Github user kambatla commented on a diff in the pull request:
https://github.com/apache/hadoop/pull/201#discussion_r108020673
--- Diff:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSSchedulerNode.java
---
@@ -38,15 +47,45 @@
public class FSSchedulerNode extends SchedulerNode {
private static final Log LOG = LogFactory.getLog(FSSchedulerNode.class);
+ private static final Comparator comparator =
+ new Comparator() {
+@Override
+public int compare(RMContainer o1, RMContainer o2) {
+ return Long.compare(o1.getContainerId().getContainerId(),
+ o2.getContainerId().getContainerId());
+}
+ };
private FSAppAttempt reservedAppSchedulable;
- private final Set containersForPreemption =
- new ConcurrentSkipListSet<>();
+ // Stores list of containers still to be preempted
+ @VisibleForTesting
+ final Set containersForPreemption =
+ new ConcurrentSkipListSet<>(comparator);
+ // Stores amount of respources preempted and reserved for each app
+ @VisibleForTesting
+ final Map
+ resourcesPreemptedForApp = new LinkedHashMap<>();
+ // Sum of resourcesPreemptedForApp values, total resources that are
+ // slated for preemption
+ Resource totalResourcesPreempted = Resource.newInstance(0, 0);
public FSSchedulerNode(RMNode node, boolean usePortForNodeName) {
super(node, usePortForNodeName);
}
+ /**
+ * Total amount of reserved resources including reservations and
preempted
+ * containers.
+ * @return total resources reserved
+ */
+ Resource getTotalReserved() {
+Resource totalReserved = getReservedContainer() != null
+? Resources.clone(getReservedContainer().getAllocatedResource())
+: Resource.newInstance(0, 0);
--- End diff --
One of the reasons I feel cloning none is better is when we add more
resources (for example through ResourceTypes).
An alternative would be to actually add a newInstance method that doesn't
take any arguments and initializes all resources (including future ones) to
zero. This could be done in a later JIRA.
> FS preemption should reserve a node before considering containers on it for
> preemption
> --
>
> Key: YARN-5829
> URL: https://issues.apache.org/jira/browse/YARN-5829
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: fairscheduler
>Reporter: Karthik Kambatla
>Assignee: Miklos Szegedi
>
> FS preemption evaluates nodes for preemption, and subsequently preempts
> identified containers. If this node is not reserved for a specific
> application, any other application could be allocated resources on this node.
> Reserving the node for the starved application before preempting containers
> would help avoid this.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5829) FS preemption should reserve a node before considering containers on it for preemption
[
https://issues.apache.org/jira/browse/YARN-5829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15941424#comment-15941424
]
ASF GitHub Bot commented on YARN-5829:
--
Github user kambatla commented on a diff in the pull request:
https://github.com/apache/hadoop/pull/201#discussion_r108023245
--- Diff:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFSSchedulerNode.java
---
@@ -0,0 +1,376 @@
+package org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair;
+
+import org.apache.hadoop.yarn.api.records.*;
+import
org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainer;
+import org.apache.hadoop.yarn.server.resourcemanager.rmnode.RMNode;
+import org.apache.hadoop.yarn.util.resource.Resources;
+import org.junit.Test;
+import org.mockito.invocation.InvocationOnMock;
+import org.mockito.stubbing.Answer;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Map;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNotEquals;
+import static org.junit.Assert.assertTrue;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+/**
+ * Test scheduler node, especially preemption reservations.
+ */
+public class TestFSSchedulerNode {
+ private long containerNum = 0;
+ private ArrayList containers = new ArrayList<>();
+
+ private RMNode createNode() {
+RMNode node = mock(RMNode.class);
+when(node.getTotalCapability()).thenReturn(Resource.newInstance(8192,
8));
+when(node.getHostName()).thenReturn("host.domain.com");
+return node;
+ }
+
+ private RMContainer createDefaultContainer() {
+return createContainer(Resource.newInstance(1024, 1), null);
+ }
+
+ private RMContainer createContainer(
+ Resource request, ApplicationAttemptId appAttemptId) {
+RMContainer container = mock(RMContainer.class);
+Container containerInner = mock(Container.class);
+ContainerId id = mock(ContainerId.class);
+when(id.getContainerId()).thenReturn(containerNum);
+when(containerInner.getResource()).
+thenReturn(Resources.clone(request));
+when(containerInner.getId()).thenReturn(id);
+when(containerInner.getExecutionType()).
+thenReturn(ExecutionType.GUARANTEED);
+when(container.getApplicationAttemptId()).thenReturn(appAttemptId);
+when(container.getContainerId()).thenReturn(id);
+when(container.getContainer()).thenReturn(containerInner);
+
when(container.getExecutionType()).thenReturn(ExecutionType.GUARANTEED);
--- End diff --
execution type is repeated. Also, do we need to add a when for all these?
> FS preemption should reserve a node before considering containers on it for
> preemption
> --
>
> Key: YARN-5829
> URL: https://issues.apache.org/jira/browse/YARN-5829
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: fairscheduler
>Reporter: Karthik Kambatla
>Assignee: Miklos Szegedi
>
> FS preemption evaluates nodes for preemption, and subsequently preempts
> identified containers. If this node is not reserved for a specific
> application, any other application could be allocated resources on this node.
> Reserving the node for the starved application before preempting containers
> would help avoid this.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5829) FS preemption should reserve a node before considering containers on it for preemption
[
https://issues.apache.org/jira/browse/YARN-5829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15941417#comment-15941417
]
ASF GitHub Bot commented on YARN-5829:
--
Github user kambatla commented on a diff in the pull request:
https://github.com/apache/hadoop/pull/201#discussion_r108021344
--- Diff:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairScheduler.java
---
@@ -968,6 +966,32 @@ private boolean shouldContinueAssigning(int containers,
}
}
+ /**
+ * Assign preempted containers to the applications that have reserved
+ * resources for preempted containers.
+ * @param node Node to check
+ * @return assignment has occurred
+ */
+ static boolean assignPreemptedContainers(FSSchedulerNode node ) {
+boolean assignedAny = false;
+node.cleanupPreemptionList();
+for (Entry entry :
+node.getPreemptionList().entrySet()) {
+ FSAppAttempt app = entry.getKey();
+ Resource reserved = Resources.clone(entry.getValue());
+ while (!app.isStopped() && !Resources.isNone(reserved)) {
+Resource assigned = app.assignContainer(node);
+if (Resources.isNone(assigned)) {
+ break;
--- End diff --
Add a comment as to why we are breaking here.
> FS preemption should reserve a node before considering containers on it for
> preemption
> --
>
> Key: YARN-5829
> URL: https://issues.apache.org/jira/browse/YARN-5829
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: fairscheduler
>Reporter: Karthik Kambatla
>Assignee: Miklos Szegedi
>
> FS preemption evaluates nodes for preemption, and subsequently preempts
> identified containers. If this node is not reserved for a specific
> application, any other application could be allocated resources on this node.
> Reserving the node for the starved application before preempting containers
> would help avoid this.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5829) FS preemption should reserve a node before considering containers on it for preemption
[
https://issues.apache.org/jira/browse/YARN-5829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15941423#comment-15941423
]
ASF GitHub Bot commented on YARN-5829:
--
Github user kambatla commented on a diff in the pull request:
https://github.com/apache/hadoop/pull/201#discussion_r108023563
--- Diff:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFSSchedulerNode.java
---
@@ -0,0 +1,376 @@
+package org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair;
+
+import org.apache.hadoop.yarn.api.records.*;
+import
org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainer;
+import org.apache.hadoop.yarn.server.resourcemanager.rmnode.RMNode;
+import org.apache.hadoop.yarn.util.resource.Resources;
+import org.junit.Test;
+import org.mockito.invocation.InvocationOnMock;
+import org.mockito.stubbing.Answer;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Map;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNotEquals;
+import static org.junit.Assert.assertTrue;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+/**
+ * Test scheduler node, especially preemption reservations.
+ */
+public class TestFSSchedulerNode {
+ private long containerNum = 0;
+ private ArrayList containers = new ArrayList<>();
+
+ private RMNode createNode() {
+RMNode node = mock(RMNode.class);
+when(node.getTotalCapability()).thenReturn(Resource.newInstance(8192,
8));
+when(node.getHostName()).thenReturn("host.domain.com");
+return node;
+ }
+
+ private RMContainer createDefaultContainer() {
+return createContainer(Resource.newInstance(1024, 1), null);
+ }
+
+ private RMContainer createContainer(
+ Resource request, ApplicationAttemptId appAttemptId) {
+RMContainer container = mock(RMContainer.class);
+Container containerInner = mock(Container.class);
+ContainerId id = mock(ContainerId.class);
+when(id.getContainerId()).thenReturn(containerNum);
+when(containerInner.getResource()).
+thenReturn(Resources.clone(request));
+when(containerInner.getId()).thenReturn(id);
+when(containerInner.getExecutionType()).
+thenReturn(ExecutionType.GUARANTEED);
+when(container.getApplicationAttemptId()).thenReturn(appAttemptId);
+when(container.getContainerId()).thenReturn(id);
+when(container.getContainer()).thenReturn(containerInner);
+
when(container.getExecutionType()).thenReturn(ExecutionType.GUARANTEED);
+when(container.getAllocatedResource()).
+thenReturn(Resources.clone(request));
+containers.add(container);
+containerNum++;
+return container;
+ }
+
+ private void saturateCluster(FSSchedulerNode schedulerNode) {
+while (!Resources.isNone(schedulerNode.getUnallocatedResource())) {
+ createDefaultContainer();
+ schedulerNode.allocateContainer(containers.get((int)containerNum -
1));
+ schedulerNode.containerStarted(containers.get((int)containerNum - 1).
+ getContainerId());
+}
+ }
+
+ private FSAppAttempt createStarvingApp(FSSchedulerNode schedulerNode,
+ Resource request) {
+FSAppAttempt starvingApp = mock(FSAppAttempt.class);
+ApplicationAttemptId appAttemptId =
+mock(ApplicationAttemptId.class);
+when(starvingApp.getApplicationAttemptId()).thenReturn(appAttemptId);
+when(starvingApp.assignContainer(schedulerNode)).thenAnswer(
+new Answer() {
+ @Override
+ public Resource answer(InvocationOnMock invocationOnMock)
+ throws Throwable {
+Resource response = Resource.newInstance(0, 0);
+while (!Resources.isNone(request) &&
+!Resources.isNone(schedulerNode.getUnallocatedResource()))
{
+ RMContainer container = createContainer(request,
appAttemptId);
+ schedulerNode.allocateContainer(container);
+ Resources.addTo(response, container.getAllocatedResource());
+ Resources.subtractFrom(request,
+ container.getAllocatedResource());
+}
+return response;
+ }
+});
+when(starvingApp.getPendingDemand()).thenReturn(request);
+return starvingApp;
+ }
+
+ private void finalValidation(FSSchedulerNode schedulerNode) {
+assertEquals("Everything should
[jira] [Commented] (YARN-5829) FS preemption should reserve a node before considering containers on it for preemption
[ https://issues.apache.org/jira/browse/YARN-5829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15941418#comment-15941418 ] ASF GitHub Bot commented on YARN-5829: -- Github user kambatla commented on a diff in the pull request: https://github.com/apache/hadoop/pull/201#discussion_r108020399 --- Diff: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSSchedulerNode.java --- @@ -18,18 +18,27 @@ package org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair; +import com.google.common.annotations.VisibleForTesting; import org.apache.commons.logging.Log; import org.apache.commons.logging.LogFactory; import org.apache.hadoop.classification.InterfaceAudience.Private; import org.apache.hadoop.classification.InterfaceStability.Unstable; import org.apache.hadoop.yarn.api.records.ApplicationAttemptId; +import org.apache.hadoop.yarn.api.records.ContainerId; +import org.apache.hadoop.yarn.api.records.Resource; import org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainer; import org.apache.hadoop.yarn.server.resourcemanager.rmnode.RMNode; import org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerApplicationAttempt; import org.apache.hadoop.yarn.server.scheduler.SchedulerRequestKey; import org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerNode; +import org.apache.hadoop.yarn.util.resource.Resources; import java.util.Collection; +import java.util.Comparator; +import java.util.Iterator; +import java.util.LinkedHashMap; +import java.util.LinkedHashSet; --- End diff -- Don't think we are using this import anymore. > FS preemption should reserve a node before considering containers on it for > preemption > -- > > Key: YARN-5829 > URL: https://issues.apache.org/jira/browse/YARN-5829 > Project: Hadoop YARN > Issue Type: Sub-task > Components: fairscheduler >Reporter: Karthik Kambatla >Assignee: Miklos Szegedi > > FS preemption evaluates nodes for preemption, and subsequently preempts > identified containers. If this node is not reserved for a specific > application, any other application could be allocated resources on this node. > Reserving the node for the starved application before preempting containers > would help avoid this. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5829) FS preemption should reserve a node before considering containers on it for preemption
[
https://issues.apache.org/jira/browse/YARN-5829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15941434#comment-15941434
]
ASF GitHub Bot commented on YARN-5829:
--
Github user kambatla commented on a diff in the pull request:
https://github.com/apache/hadoop/pull/201#discussion_r108020477
--- Diff:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSSchedulerNode.java
---
@@ -38,15 +47,45 @@
public class FSSchedulerNode extends SchedulerNode {
private static final Log LOG = LogFactory.getLog(FSSchedulerNode.class);
+ private static final Comparator comparator =
+ new Comparator() {
+@Override
+public int compare(RMContainer o1, RMContainer o2) {
+ return Long.compare(o1.getContainerId().getContainerId(),
+ o2.getContainerId().getContainerId());
+}
+ };
private FSAppAttempt reservedAppSchedulable;
- private final Set containersForPreemption =
- new ConcurrentSkipListSet<>();
+ // Stores list of containers still to be preempted
+ @VisibleForTesting
+ final Set containersForPreemption =
+ new ConcurrentSkipListSet<>(comparator);
+ // Stores amount of respources preempted and reserved for each app
--- End diff --
Typo in resources
> FS preemption should reserve a node before considering containers on it for
> preemption
> --
>
> Key: YARN-5829
> URL: https://issues.apache.org/jira/browse/YARN-5829
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: fairscheduler
>Reporter: Karthik Kambatla
>Assignee: Miklos Szegedi
>
> FS preemption evaluates nodes for preemption, and subsequently preempts
> identified containers. If this node is not reserved for a specific
> application, any other application could be allocated resources on this node.
> Reserving the node for the starved application before preempting containers
> would help avoid this.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5829) FS preemption should reserve a node before considering containers on it for preemption
[
https://issues.apache.org/jira/browse/YARN-5829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15941437#comment-15941437
]
ASF GitHub Bot commented on YARN-5829:
--
Github user kambatla commented on a diff in the pull request:
https://github.com/apache/hadoop/pull/201#discussion_r108023522
--- Diff:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFSSchedulerNode.java
---
@@ -0,0 +1,376 @@
+package org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair;
+
+import org.apache.hadoop.yarn.api.records.*;
+import
org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainer;
+import org.apache.hadoop.yarn.server.resourcemanager.rmnode.RMNode;
+import org.apache.hadoop.yarn.util.resource.Resources;
+import org.junit.Test;
+import org.mockito.invocation.InvocationOnMock;
+import org.mockito.stubbing.Answer;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Map;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNotEquals;
+import static org.junit.Assert.assertTrue;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+/**
+ * Test scheduler node, especially preemption reservations.
+ */
+public class TestFSSchedulerNode {
+ private long containerNum = 0;
+ private ArrayList containers = new ArrayList<>();
+
+ private RMNode createNode() {
+RMNode node = mock(RMNode.class);
+when(node.getTotalCapability()).thenReturn(Resource.newInstance(8192,
8));
+when(node.getHostName()).thenReturn("host.domain.com");
+return node;
+ }
+
+ private RMContainer createDefaultContainer() {
+return createContainer(Resource.newInstance(1024, 1), null);
+ }
+
+ private RMContainer createContainer(
+ Resource request, ApplicationAttemptId appAttemptId) {
+RMContainer container = mock(RMContainer.class);
+Container containerInner = mock(Container.class);
+ContainerId id = mock(ContainerId.class);
+when(id.getContainerId()).thenReturn(containerNum);
+when(containerInner.getResource()).
+thenReturn(Resources.clone(request));
+when(containerInner.getId()).thenReturn(id);
+when(containerInner.getExecutionType()).
+thenReturn(ExecutionType.GUARANTEED);
+when(container.getApplicationAttemptId()).thenReturn(appAttemptId);
+when(container.getContainerId()).thenReturn(id);
+when(container.getContainer()).thenReturn(containerInner);
+
when(container.getExecutionType()).thenReturn(ExecutionType.GUARANTEED);
+when(container.getAllocatedResource()).
+thenReturn(Resources.clone(request));
+containers.add(container);
+containerNum++;
+return container;
+ }
+
+ private void saturateCluster(FSSchedulerNode schedulerNode) {
+while (!Resources.isNone(schedulerNode.getUnallocatedResource())) {
+ createDefaultContainer();
+ schedulerNode.allocateContainer(containers.get((int)containerNum -
1));
+ schedulerNode.containerStarted(containers.get((int)containerNum - 1).
+ getContainerId());
+}
+ }
+
+ private FSAppAttempt createStarvingApp(FSSchedulerNode schedulerNode,
+ Resource request) {
+FSAppAttempt starvingApp = mock(FSAppAttempt.class);
+ApplicationAttemptId appAttemptId =
+mock(ApplicationAttemptId.class);
+when(starvingApp.getApplicationAttemptId()).thenReturn(appAttemptId);
+when(starvingApp.assignContainer(schedulerNode)).thenAnswer(
+new Answer() {
+ @Override
+ public Resource answer(InvocationOnMock invocationOnMock)
+ throws Throwable {
+Resource response = Resource.newInstance(0, 0);
+while (!Resources.isNone(request) &&
+!Resources.isNone(schedulerNode.getUnallocatedResource()))
{
+ RMContainer container = createContainer(request,
appAttemptId);
+ schedulerNode.allocateContainer(container);
+ Resources.addTo(response, container.getAllocatedResource());
+ Resources.subtractFrom(request,
+ container.getAllocatedResource());
+}
+return response;
+ }
+});
+when(starvingApp.getPendingDemand()).thenReturn(request);
+return starvingApp;
+ }
+
+ private void finalValidation(FSSchedulerNode schedulerNode) {
+assertEquals("Everything should
[jira] [Commented] (YARN-5829) FS preemption should reserve a node before considering containers on it for preemption
[
https://issues.apache.org/jira/browse/YARN-5829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15941425#comment-15941425
]
ASF GitHub Bot commented on YARN-5829:
--
Github user kambatla commented on a diff in the pull request:
https://github.com/apache/hadoop/pull/201#discussion_r108021737
--- Diff:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSAppAttempt.java
---
@@ -646,7 +646,6 @@ private Container createContainer(FSSchedulerNode node,
Resource capability,
private boolean reserve(Resource perAllocationResource, FSSchedulerNode
node,
Container reservedContainer, NodeType type,
SchedulerRequestKey schedulerKey) {
-
--- End diff --
Spurious change.
> FS preemption should reserve a node before considering containers on it for
> preemption
> --
>
> Key: YARN-5829
> URL: https://issues.apache.org/jira/browse/YARN-5829
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: fairscheduler
>Reporter: Karthik Kambatla
>Assignee: Miklos Szegedi
>
> FS preemption evaluates nodes for preemption, and subsequently preempts
> identified containers. If this node is not reserved for a specific
> application, any other application could be allocated resources on this node.
> Reserving the node for the starved application before preempting containers
> would help avoid this.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5829) FS preemption should reserve a node before considering containers on it for preemption
[
https://issues.apache.org/jira/browse/YARN-5829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15941432#comment-15941432
]
ASF GitHub Bot commented on YARN-5829:
--
Github user kambatla commented on a diff in the pull request:
https://github.com/apache/hadoop/pull/201#discussion_r108020432
--- Diff:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSSchedulerNode.java
---
@@ -38,15 +47,45 @@
public class FSSchedulerNode extends SchedulerNode {
private static final Log LOG = LogFactory.getLog(FSSchedulerNode.class);
+ private static final Comparator comparator =
--- End diff --
If you have RMContainer extend Comparable, you don't need this comparator.
> FS preemption should reserve a node before considering containers on it for
> preemption
> --
>
> Key: YARN-5829
> URL: https://issues.apache.org/jira/browse/YARN-5829
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: fairscheduler
>Reporter: Karthik Kambatla
>Assignee: Miklos Szegedi
>
> FS preemption evaluates nodes for preemption, and subsequently preempts
> identified containers. If this node is not reserved for a specific
> application, any other application could be allocated resources on this node.
> Reserving the node for the starved application before preempting containers
> would help avoid this.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5829) FS preemption should reserve a node before considering containers on it for preemption
[
https://issues.apache.org/jira/browse/YARN-5829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15941444#comment-15941444
]
ASF GitHub Bot commented on YARN-5829:
--
Github user kambatla commented on a diff in the pull request:
https://github.com/apache/hadoop/pull/201#discussion_r108023660
--- Diff:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairSchedulerPreemption.java
---
@@ -294,11 +294,29 @@ private void verifyPreemption(int
numStarvedAppContainers)
8 - 2 * numStarvedAppContainers,
greedyApp.getQueue().getMetrics().getAggregatePreemptedContainers());
+// Verify the node is reserved for the starvingApp
+for (RMNode rmNode : rmNodes) {
+ FSSchedulerNode node = (FSSchedulerNode)
+ scheduler.getNodeTracker().getNode(rmNode.getNodeID());
+ if (node.getContainersForPreemption().size() > 0) {
+
assertTrue(node.getPreemptionList().keySet().contains(starvingApp));
+ }
+}
+
sendEnoughNodeUpdatesToAssignFully();
// Verify the preempted containers are assigned to starvingApp
assertEquals("Starved app is not assigned the right # of containers",
numStarvedAppContainers, starvingApp.getLiveContainers().size());
+
+// Verify the node is not reserved for the starvingApp anymore
+for (RMNode rmNode : rmNodes) {
+ FSSchedulerNode node = (FSSchedulerNode)
+ scheduler.getNodeTracker().getNode(rmNode.getNodeID());
+ if (node.getContainersForPreemption().size() > 0) {
+
assertTrue(!node.getPreemptionList().keySet().contains(starvingApp));
--- End diff --
assert message.
> FS preemption should reserve a node before considering containers on it for
> preemption
> --
>
> Key: YARN-5829
> URL: https://issues.apache.org/jira/browse/YARN-5829
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: fairscheduler
>Reporter: Karthik Kambatla
>Assignee: Miklos Szegedi
>
> FS preemption evaluates nodes for preemption, and subsequently preempts
> identified containers. If this node is not reserved for a specific
> application, any other application could be allocated resources on this node.
> Reserving the node for the starved application before preempting containers
> would help avoid this.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5829) FS preemption should reserve a node before considering containers on it for preemption
[
https://issues.apache.org/jira/browse/YARN-5829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15941433#comment-15941433
]
ASF GitHub Bot commented on YARN-5829:
--
Github user kambatla commented on a diff in the pull request:
https://github.com/apache/hadoop/pull/201#discussion_r108023665
--- Diff:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairSchedulerPreemption.java
---
@@ -294,11 +294,29 @@ private void verifyPreemption(int
numStarvedAppContainers)
8 - 2 * numStarvedAppContainers,
greedyApp.getQueue().getMetrics().getAggregatePreemptedContainers());
+// Verify the node is reserved for the starvingApp
+for (RMNode rmNode : rmNodes) {
+ FSSchedulerNode node = (FSSchedulerNode)
+ scheduler.getNodeTracker().getNode(rmNode.getNodeID());
+ if (node.getContainersForPreemption().size() > 0) {
+
assertTrue(node.getPreemptionList().keySet().contains(starvingApp));
--- End diff --
assert message.
> FS preemption should reserve a node before considering containers on it for
> preemption
> --
>
> Key: YARN-5829
> URL: https://issues.apache.org/jira/browse/YARN-5829
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: fairscheduler
>Reporter: Karthik Kambatla
>Assignee: Miklos Szegedi
>
> FS preemption evaluates nodes for preemption, and subsequently preempts
> identified containers. If this node is not reserved for a specific
> application, any other application could be allocated resources on this node.
> Reserving the node for the starved application before preempting containers
> would help avoid this.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5829) FS preemption should reserve a node before considering containers on it for preemption
[
https://issues.apache.org/jira/browse/YARN-5829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15941436#comment-15941436
]
ASF GitHub Bot commented on YARN-5829:
--
Github user kambatla commented on a diff in the pull request:
https://github.com/apache/hadoop/pull/201#discussion_r108021228
--- Diff:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairScheduler.java
---
@@ -968,6 +966,32 @@ private boolean shouldContinueAssigning(int containers,
}
}
+ /**
+ * Assign preempted containers to the applications that have reserved
+ * resources for preempted containers.
+ * @param node Node to check
+ * @return assignment has occurred
+ */
+ static boolean assignPreemptedContainers(FSSchedulerNode node ) {
+boolean assignedAny = false;
+node.cleanupPreemptionList();
--- End diff --
Let us rely on the call to getPreemptionList to do cleanup instead.
> FS preemption should reserve a node before considering containers on it for
> preemption
> --
>
> Key: YARN-5829
> URL: https://issues.apache.org/jira/browse/YARN-5829
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: fairscheduler
>Reporter: Karthik Kambatla
>Assignee: Miklos Szegedi
>
> FS preemption evaluates nodes for preemption, and subsequently preempts
> identified containers. If this node is not reserved for a specific
> application, any other application could be allocated resources on this node.
> Reserving the node for the starved application before preempting containers
> would help avoid this.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5829) FS preemption should reserve a node before considering containers on it for preemption
[
https://issues.apache.org/jira/browse/YARN-5829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15941445#comment-15941445
]
ASF GitHub Bot commented on YARN-5829:
--
Github user kambatla commented on a diff in the pull request:
https://github.com/apache/hadoop/pull/201#discussion_r108023213
--- Diff:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFSSchedulerNode.java
---
@@ -0,0 +1,376 @@
+package org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair;
+
+import org.apache.hadoop.yarn.api.records.*;
+import
org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainer;
+import org.apache.hadoop.yarn.server.resourcemanager.rmnode.RMNode;
+import org.apache.hadoop.yarn.util.resource.Resources;
+import org.junit.Test;
+import org.mockito.invocation.InvocationOnMock;
+import org.mockito.stubbing.Answer;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Map;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNotEquals;
+import static org.junit.Assert.assertTrue;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+/**
+ * Test scheduler node, especially preemption reservations.
+ */
+public class TestFSSchedulerNode {
+ private long containerNum = 0;
+ private ArrayList containers = new ArrayList<>();
--- End diff --
Should the arraylist be final?
> FS preemption should reserve a node before considering containers on it for
> preemption
> --
>
> Key: YARN-5829
> URL: https://issues.apache.org/jira/browse/YARN-5829
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: fairscheduler
>Reporter: Karthik Kambatla
>Assignee: Miklos Szegedi
>
> FS preemption evaluates nodes for preemption, and subsequently preempts
> identified containers. If this node is not reserved for a specific
> application, any other application could be allocated resources on this node.
> Reserving the node for the starved application before preempting containers
> would help avoid this.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5829) FS preemption should reserve a node before considering containers on it for preemption
[
https://issues.apache.org/jira/browse/YARN-5829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15941438#comment-15941438
]
ASF GitHub Bot commented on YARN-5829:
--
Github user kambatla commented on a diff in the pull request:
https://github.com/apache/hadoop/pull/201#discussion_r108020976
--- Diff:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSSchedulerNode.java
---
@@ -110,16 +149,57 @@ synchronized FSAppAttempt getReservedAppSchedulable()
{
}
/**
+ * List reserved resources after preemption and assign them to the
+ * appropriate applications in a FIFO order.
+ * It is a good practice to call {@link #cleanupPreemptionList()}
+ * after this call
+ * @return if any resources were allocated
+ */
+ synchronized LinkedHashMap getPreemptionList() {
+cleanupPreemptionList();
+return new LinkedHashMap<>(resourcesPreemptedForApp);
+ }
+
+ /**
+ * Remove apps that have their preemption requests fulfilled
+ */
+ synchronized void cleanupPreemptionList() {
+Iterator iterator =
+resourcesPreemptedForApp.keySet().iterator();
+while (iterator.hasNext()) {
+ FSAppAttempt app = iterator.next();
+ boolean removeApp = false;
+ if (app.isStopped() || Resources.equals(
+ app.getPendingDemand(), Resources.none())) {
+// App does not need more resources
+Resources.subtractFrom(totalResourcesPreempted,
+resourcesPreemptedForApp.get(app));
+iterator.remove();
+ }
+}
+ }
+
+ /**
* Mark {@code containers} as being considered for preemption so they are
* not considered again. A call to this requires a corresponding call to
- * {@link #removeContainerForPreemption} to ensure we do not mark a
+ * removeContainer for preemption to ensure we do not mark a
--- End diff --
Needs javadoc fixing.
> FS preemption should reserve a node before considering containers on it for
> preemption
> --
>
> Key: YARN-5829
> URL: https://issues.apache.org/jira/browse/YARN-5829
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: fairscheduler
>Reporter: Karthik Kambatla
>Assignee: Miklos Szegedi
>
> FS preemption evaluates nodes for preemption, and subsequently preempts
> identified containers. If this node is not reserved for a specific
> application, any other application could be allocated resources on this node.
> Reserving the node for the starved application before preempting containers
> would help avoid this.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5829) FS preemption should reserve a node before considering containers on it for preemption
[
https://issues.apache.org/jira/browse/YARN-5829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15941431#comment-15941431
]
ASF GitHub Bot commented on YARN-5829:
--
Github user kambatla commented on a diff in the pull request:
https://github.com/apache/hadoop/pull/201#discussion_r108023650
--- Diff:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairSchedulerPreemption.java
---
@@ -294,11 +294,29 @@ private void verifyPreemption(int
numStarvedAppContainers)
8 - 2 * numStarvedAppContainers,
greedyApp.getQueue().getMetrics().getAggregatePreemptedContainers());
+// Verify the node is reserved for the starvingApp
+for (RMNode rmNode : rmNodes) {
+ FSSchedulerNode node = (FSSchedulerNode)
+ scheduler.getNodeTracker().getNode(rmNode.getNodeID());
+ if (node.getContainersForPreemption().size() > 0) {
+
assertTrue(node.getPreemptionList().keySet().contains(starvingApp));
+ }
+}
+
sendEnoughNodeUpdatesToAssignFully();
// Verify the preempted containers are assigned to starvingApp
assertEquals("Starved app is not assigned the right # of containers",
numStarvedAppContainers, starvingApp.getLiveContainers().size());
+
+// Verify the node is not reserved for the starvingApp anymore
+for (RMNode rmNode : rmNodes) {
+ FSSchedulerNode node = (FSSchedulerNode)
+ scheduler.getNodeTracker().getNode(rmNode.getNodeID());
+ if (node.getContainersForPreemption().size() > 0) {
+
assertTrue(!node.getPreemptionList().keySet().contains(starvingApp));
--- End diff --
assertFalse?
> FS preemption should reserve a node before considering containers on it for
> preemption
> --
>
> Key: YARN-5829
> URL: https://issues.apache.org/jira/browse/YARN-5829
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: fairscheduler
>Reporter: Karthik Kambatla
>Assignee: Miklos Szegedi
>
> FS preemption evaluates nodes for preemption, and subsequently preempts
> identified containers. If this node is not reserved for a specific
> application, any other application could be allocated resources on this node.
> Reserving the node for the starved application before preempting containers
> would help avoid this.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5829) FS preemption should reserve a node before considering containers on it for preemption
[
https://issues.apache.org/jira/browse/YARN-5829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15941439#comment-15941439
]
ASF GitHub Bot commented on YARN-5829:
--
Github user kambatla commented on a diff in the pull request:
https://github.com/apache/hadoop/pull/201#discussion_r108020913
--- Diff:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSSchedulerNode.java
---
@@ -110,16 +149,57 @@ synchronized FSAppAttempt getReservedAppSchedulable()
{
}
/**
+ * List reserved resources after preemption and assign them to the
+ * appropriate applications in a FIFO order.
+ * It is a good practice to call {@link #cleanupPreemptionList()}
+ * after this call
+ * @return if any resources were allocated
+ */
+ synchronized LinkedHashMap getPreemptionList() {
+cleanupPreemptionList();
+return new LinkedHashMap<>(resourcesPreemptedForApp);
+ }
+
+ /**
+ * Remove apps that have their preemption requests fulfilled
+ */
+ synchronized void cleanupPreemptionList() {
+Iterator iterator =
+resourcesPreemptedForApp.keySet().iterator();
+while (iterator.hasNext()) {
+ FSAppAttempt app = iterator.next();
+ boolean removeApp = false;
+ if (app.isStopped() || Resources.equals(
--- End diff --
Use Resources.isNone instead.
> FS preemption should reserve a node before considering containers on it for
> preemption
> --
>
> Key: YARN-5829
> URL: https://issues.apache.org/jira/browse/YARN-5829
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: fairscheduler
>Reporter: Karthik Kambatla
>Assignee: Miklos Szegedi
>
> FS preemption evaluates nodes for preemption, and subsequently preempts
> identified containers. If this node is not reserved for a specific
> application, any other application could be allocated resources on this node.
> Reserving the node for the starved application before preempting containers
> would help avoid this.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5829) FS preemption should reserve a node before considering containers on it for preemption
[
https://issues.apache.org/jira/browse/YARN-5829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15941430#comment-15941430
]
ASF GitHub Bot commented on YARN-5829:
--
Github user kambatla commented on a diff in the pull request:
https://github.com/apache/hadoop/pull/201#discussion_r108023020
--- Diff:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSSchedulerNode.java
---
@@ -110,16 +149,57 @@ synchronized FSAppAttempt getReservedAppSchedulable()
{
}
/**
+ * List reserved resources after preemption and assign them to the
+ * appropriate applications in a FIFO order.
+ * It is a good practice to call {@link #cleanupPreemptionList()}
+ * after this call
+ * @return if any resources were allocated
+ */
+ synchronized LinkedHashMap getPreemptionList() {
+cleanupPreemptionList();
+return new LinkedHashMap<>(resourcesPreemptedForApp);
+ }
+
+ /**
+ * Remove apps that have their preemption requests fulfilled
+ */
+ synchronized void cleanupPreemptionList() {
+Iterator iterator =
+resourcesPreemptedForApp.keySet().iterator();
+while (iterator.hasNext()) {
+ FSAppAttempt app = iterator.next();
+ boolean removeApp = false;
--- End diff --
This variable is not used.
> FS preemption should reserve a node before considering containers on it for
> preemption
> --
>
> Key: YARN-5829
> URL: https://issues.apache.org/jira/browse/YARN-5829
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: fairscheduler
>Reporter: Karthik Kambatla
>Assignee: Miklos Szegedi
>
> FS preemption evaluates nodes for preemption, and subsequently preempts
> identified containers. If this node is not reserved for a specific
> application, any other application could be allocated resources on this node.
> Reserving the node for the starved application before preempting containers
> would help avoid this.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5829) FS preemption should reserve a node before considering containers on it for preemption
[
https://issues.apache.org/jira/browse/YARN-5829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15941427#comment-15941427
]
ASF GitHub Bot commented on YARN-5829:
--
Github user kambatla commented on a diff in the pull request:
https://github.com/apache/hadoop/pull/201#discussion_r108023320
--- Diff:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFSSchedulerNode.java
---
@@ -0,0 +1,376 @@
+package org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair;
+
+import org.apache.hadoop.yarn.api.records.*;
+import
org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainer;
+import org.apache.hadoop.yarn.server.resourcemanager.rmnode.RMNode;
+import org.apache.hadoop.yarn.util.resource.Resources;
+import org.junit.Test;
+import org.mockito.invocation.InvocationOnMock;
+import org.mockito.stubbing.Answer;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Map;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNotEquals;
+import static org.junit.Assert.assertTrue;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+/**
+ * Test scheduler node, especially preemption reservations.
+ */
+public class TestFSSchedulerNode {
+ private long containerNum = 0;
+ private ArrayList containers = new ArrayList<>();
+
+ private RMNode createNode() {
+RMNode node = mock(RMNode.class);
+when(node.getTotalCapability()).thenReturn(Resource.newInstance(8192,
8));
+when(node.getHostName()).thenReturn("host.domain.com");
+return node;
+ }
+
+ private RMContainer createDefaultContainer() {
+return createContainer(Resource.newInstance(1024, 1), null);
+ }
+
+ private RMContainer createContainer(
+ Resource request, ApplicationAttemptId appAttemptId) {
+RMContainer container = mock(RMContainer.class);
+Container containerInner = mock(Container.class);
+ContainerId id = mock(ContainerId.class);
+when(id.getContainerId()).thenReturn(containerNum);
+when(containerInner.getResource()).
+thenReturn(Resources.clone(request));
+when(containerInner.getId()).thenReturn(id);
+when(containerInner.getExecutionType()).
+thenReturn(ExecutionType.GUARANTEED);
+when(container.getApplicationAttemptId()).thenReturn(appAttemptId);
+when(container.getContainerId()).thenReturn(id);
+when(container.getContainer()).thenReturn(containerInner);
+
when(container.getExecutionType()).thenReturn(ExecutionType.GUARANTEED);
+when(container.getAllocatedResource()).
+thenReturn(Resources.clone(request));
+containers.add(container);
+containerNum++;
+return container;
+ }
+
+ private void saturateCluster(FSSchedulerNode schedulerNode) {
+while (!Resources.isNone(schedulerNode.getUnallocatedResource())) {
+ createDefaultContainer();
+ schedulerNode.allocateContainer(containers.get((int)containerNum -
1));
--- End diff --
You don't need to typecast if you change it to int.
> FS preemption should reserve a node before considering containers on it for
> preemption
> --
>
> Key: YARN-5829
> URL: https://issues.apache.org/jira/browse/YARN-5829
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: fairscheduler
>Reporter: Karthik Kambatla
>Assignee: Miklos Szegedi
>
> FS preemption evaluates nodes for preemption, and subsequently preempts
> identified containers. If this node is not reserved for a specific
> application, any other application could be allocated resources on this node.
> Reserving the node for the starved application before preempting containers
> would help avoid this.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5829) FS preemption should reserve a node before considering containers on it for preemption
[
https://issues.apache.org/jira/browse/YARN-5829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15941429#comment-15941429
]
ASF GitHub Bot commented on YARN-5829:
--
Github user kambatla commented on a diff in the pull request:
https://github.com/apache/hadoop/pull/201#discussion_r108023440
--- Diff:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFSSchedulerNode.java
---
@@ -0,0 +1,376 @@
+package org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair;
+
+import org.apache.hadoop.yarn.api.records.*;
+import
org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainer;
+import org.apache.hadoop.yarn.server.resourcemanager.rmnode.RMNode;
+import org.apache.hadoop.yarn.util.resource.Resources;
+import org.junit.Test;
+import org.mockito.invocation.InvocationOnMock;
+import org.mockito.stubbing.Answer;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Map;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNotEquals;
+import static org.junit.Assert.assertTrue;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+/**
+ * Test scheduler node, especially preemption reservations.
+ */
+public class TestFSSchedulerNode {
+ private long containerNum = 0;
+ private ArrayList containers = new ArrayList<>();
+
+ private RMNode createNode() {
+RMNode node = mock(RMNode.class);
+when(node.getTotalCapability()).thenReturn(Resource.newInstance(8192,
8));
+when(node.getHostName()).thenReturn("host.domain.com");
+return node;
+ }
+
+ private RMContainer createDefaultContainer() {
+return createContainer(Resource.newInstance(1024, 1), null);
+ }
+
+ private RMContainer createContainer(
+ Resource request, ApplicationAttemptId appAttemptId) {
+RMContainer container = mock(RMContainer.class);
+Container containerInner = mock(Container.class);
+ContainerId id = mock(ContainerId.class);
+when(id.getContainerId()).thenReturn(containerNum);
+when(containerInner.getResource()).
+thenReturn(Resources.clone(request));
+when(containerInner.getId()).thenReturn(id);
+when(containerInner.getExecutionType()).
+thenReturn(ExecutionType.GUARANTEED);
+when(container.getApplicationAttemptId()).thenReturn(appAttemptId);
+when(container.getContainerId()).thenReturn(id);
+when(container.getContainer()).thenReturn(containerInner);
+
when(container.getExecutionType()).thenReturn(ExecutionType.GUARANTEED);
+when(container.getAllocatedResource()).
+thenReturn(Resources.clone(request));
+containers.add(container);
+containerNum++;
+return container;
+ }
+
+ private void saturateCluster(FSSchedulerNode schedulerNode) {
+while (!Resources.isNone(schedulerNode.getUnallocatedResource())) {
+ createDefaultContainer();
+ schedulerNode.allocateContainer(containers.get((int)containerNum -
1));
+ schedulerNode.containerStarted(containers.get((int)containerNum - 1).
+ getContainerId());
+}
+ }
+
+ private FSAppAttempt createStarvingApp(FSSchedulerNode schedulerNode,
+ Resource request) {
+FSAppAttempt starvingApp = mock(FSAppAttempt.class);
+ApplicationAttemptId appAttemptId =
+mock(ApplicationAttemptId.class);
+when(starvingApp.getApplicationAttemptId()).thenReturn(appAttemptId);
+when(starvingApp.assignContainer(schedulerNode)).thenAnswer(
+new Answer() {
+ @Override
+ public Resource answer(InvocationOnMock invocationOnMock)
+ throws Throwable {
+Resource response = Resource.newInstance(0, 0);
+while (!Resources.isNone(request) &&
+!Resources.isNone(schedulerNode.getUnallocatedResource()))
{
+ RMContainer container = createContainer(request,
appAttemptId);
+ schedulerNode.allocateContainer(container);
+ Resources.addTo(response, container.getAllocatedResource());
+ Resources.subtractFrom(request,
+ container.getAllocatedResource());
+}
+return response;
+ }
+});
+when(starvingApp.getPendingDemand()).thenReturn(request);
+return starvingApp;
+ }
+
+ private void finalValidation(FSSchedulerNode schedulerNode) {
+assertEquals("Everything should
[jira] [Commented] (YARN-5829) FS preemption should reserve a node before considering containers on it for preemption
[
https://issues.apache.org/jira/browse/YARN-5829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15941443#comment-15941443
]
ASF GitHub Bot commented on YARN-5829:
--
Github user kambatla commented on a diff in the pull request:
https://github.com/apache/hadoop/pull/201#discussion_r108020845
--- Diff:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSSchedulerNode.java
---
@@ -110,16 +149,57 @@ synchronized FSAppAttempt getReservedAppSchedulable()
{
}
/**
+ * List reserved resources after preemption and assign them to the
+ * appropriate applications in a FIFO order.
+ * It is a good practice to call {@link #cleanupPreemptionList()}
+ * after this call
+ * @return if any resources were allocated
+ */
+ synchronized LinkedHashMap getPreemptionList() {
+cleanupPreemptionList();
+return new LinkedHashMap<>(resourcesPreemptedForApp);
+ }
+
+ /**
+ * Remove apps that have their preemption requests fulfilled
+ */
+ synchronized void cleanupPreemptionList() {
--- End diff --
This method should be private. Offline, you mentioned this is called in
tests. I would also see if that can be avoided.
> FS preemption should reserve a node before considering containers on it for
> preemption
> --
>
> Key: YARN-5829
> URL: https://issues.apache.org/jira/browse/YARN-5829
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: fairscheduler
>Reporter: Karthik Kambatla
>Assignee: Miklos Szegedi
>
> FS preemption evaluates nodes for preemption, and subsequently preempts
> identified containers. If this node is not reserved for a specific
> application, any other application could be allocated resources on this node.
> Reserving the node for the starved application before preempting containers
> would help avoid this.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5829) FS preemption should reserve a node before considering containers on it for preemption
[
https://issues.apache.org/jira/browse/YARN-5829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15941442#comment-15941442
]
ASF GitHub Bot commented on YARN-5829:
--
Github user kambatla commented on a diff in the pull request:
https://github.com/apache/hadoop/pull/201#discussion_r108023192
--- Diff:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFSSchedulerNode.java
---
@@ -0,0 +1,376 @@
+package org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair;
+
+import org.apache.hadoop.yarn.api.records.*;
+import
org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainer;
+import org.apache.hadoop.yarn.server.resourcemanager.rmnode.RMNode;
+import org.apache.hadoop.yarn.util.resource.Resources;
+import org.junit.Test;
+import org.mockito.invocation.InvocationOnMock;
+import org.mockito.stubbing.Answer;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Map;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNotEquals;
+import static org.junit.Assert.assertTrue;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+/**
+ * Test scheduler node, especially preemption reservations.
+ */
+public class TestFSSchedulerNode {
+ private long containerNum = 0;
--- End diff --
For this test, do you really need a long?
> FS preemption should reserve a node before considering containers on it for
> preemption
> --
>
> Key: YARN-5829
> URL: https://issues.apache.org/jira/browse/YARN-5829
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: fairscheduler
>Reporter: Karthik Kambatla
>Assignee: Miklos Szegedi
>
> FS preemption evaluates nodes for preemption, and subsequently preempts
> identified containers. If this node is not reserved for a specific
> application, any other application could be allocated resources on this node.
> Reserving the node for the starved application before preempting containers
> would help avoid this.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5829) FS preemption should reserve a node before considering containers on it for preemption
[
https://issues.apache.org/jira/browse/YARN-5829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15941428#comment-15941428
]
ASF GitHub Bot commented on YARN-5829:
--
Github user kambatla commented on a diff in the pull request:
https://github.com/apache/hadoop/pull/201#discussion_r108023612
--- Diff:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFSSchedulerNode.java
---
@@ -0,0 +1,376 @@
+package org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair;
+
+import org.apache.hadoop.yarn.api.records.*;
+import
org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainer;
+import org.apache.hadoop.yarn.server.resourcemanager.rmnode.RMNode;
+import org.apache.hadoop.yarn.util.resource.Resources;
+import org.junit.Test;
+import org.mockito.invocation.InvocationOnMock;
+import org.mockito.stubbing.Answer;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Map;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNotEquals;
+import static org.junit.Assert.assertTrue;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+/**
+ * Test scheduler node, especially preemption reservations.
+ */
+public class TestFSSchedulerNode {
+ private long containerNum = 0;
+ private ArrayList containers = new ArrayList<>();
+
+ private RMNode createNode() {
+RMNode node = mock(RMNode.class);
+when(node.getTotalCapability()).thenReturn(Resource.newInstance(8192,
8));
+when(node.getHostName()).thenReturn("host.domain.com");
+return node;
+ }
+
+ private RMContainer createDefaultContainer() {
+return createContainer(Resource.newInstance(1024, 1), null);
+ }
+
+ private RMContainer createContainer(
+ Resource request, ApplicationAttemptId appAttemptId) {
+RMContainer container = mock(RMContainer.class);
+Container containerInner = mock(Container.class);
+ContainerId id = mock(ContainerId.class);
+when(id.getContainerId()).thenReturn(containerNum);
+when(containerInner.getResource()).
+thenReturn(Resources.clone(request));
+when(containerInner.getId()).thenReturn(id);
+when(containerInner.getExecutionType()).
+thenReturn(ExecutionType.GUARANTEED);
+when(container.getApplicationAttemptId()).thenReturn(appAttemptId);
+when(container.getContainerId()).thenReturn(id);
+when(container.getContainer()).thenReturn(containerInner);
+
when(container.getExecutionType()).thenReturn(ExecutionType.GUARANTEED);
+when(container.getAllocatedResource()).
+thenReturn(Resources.clone(request));
+containers.add(container);
+containerNum++;
+return container;
+ }
+
+ private void saturateCluster(FSSchedulerNode schedulerNode) {
+while (!Resources.isNone(schedulerNode.getUnallocatedResource())) {
+ createDefaultContainer();
+ schedulerNode.allocateContainer(containers.get((int)containerNum -
1));
+ schedulerNode.containerStarted(containers.get((int)containerNum - 1).
+ getContainerId());
+}
+ }
+
+ private FSAppAttempt createStarvingApp(FSSchedulerNode schedulerNode,
+ Resource request) {
+FSAppAttempt starvingApp = mock(FSAppAttempt.class);
+ApplicationAttemptId appAttemptId =
+mock(ApplicationAttemptId.class);
+when(starvingApp.getApplicationAttemptId()).thenReturn(appAttemptId);
+when(starvingApp.assignContainer(schedulerNode)).thenAnswer(
+new Answer() {
+ @Override
+ public Resource answer(InvocationOnMock invocationOnMock)
+ throws Throwable {
+Resource response = Resource.newInstance(0, 0);
+while (!Resources.isNone(request) &&
+!Resources.isNone(schedulerNode.getUnallocatedResource()))
{
+ RMContainer container = createContainer(request,
appAttemptId);
+ schedulerNode.allocateContainer(container);
+ Resources.addTo(response, container.getAllocatedResource());
+ Resources.subtractFrom(request,
+ container.getAllocatedResource());
+}
+return response;
+ }
+});
+when(starvingApp.getPendingDemand()).thenReturn(request);
+return starvingApp;
+ }
+
+ private void finalValidation(FSSchedulerNode schedulerNode) {
+assertEquals("Everything should
[jira] [Commented] (YARN-5829) FS preemption should reserve a node before considering containers on it for preemption
[
https://issues.apache.org/jira/browse/YARN-5829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15941420#comment-15941420
]
ASF GitHub Bot commented on YARN-5829:
--
Github user kambatla commented on a diff in the pull request:
https://github.com/apache/hadoop/pull/201#discussion_r108023329
--- Diff:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFSSchedulerNode.java
---
@@ -0,0 +1,376 @@
+package org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair;
+
+import org.apache.hadoop.yarn.api.records.*;
+import
org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainer;
+import org.apache.hadoop.yarn.server.resourcemanager.rmnode.RMNode;
+import org.apache.hadoop.yarn.util.resource.Resources;
+import org.junit.Test;
+import org.mockito.invocation.InvocationOnMock;
+import org.mockito.stubbing.Answer;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Map;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNotEquals;
+import static org.junit.Assert.assertTrue;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+/**
+ * Test scheduler node, especially preemption reservations.
+ */
+public class TestFSSchedulerNode {
+ private long containerNum = 0;
--- End diff --
Also, numContainers might be a better name.
> FS preemption should reserve a node before considering containers on it for
> preemption
> --
>
> Key: YARN-5829
> URL: https://issues.apache.org/jira/browse/YARN-5829
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: fairscheduler
>Reporter: Karthik Kambatla
>Assignee: Miklos Szegedi
>
> FS preemption evaluates nodes for preemption, and subsequently preempts
> identified containers. If this node is not reserved for a specific
> application, any other application could be allocated resources on this node.
> Reserving the node for the starved application before preempting containers
> would help avoid this.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5829) FS preemption should reserve a node before considering containers on it for preemption
[
https://issues.apache.org/jira/browse/YARN-5829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15941422#comment-15941422
]
ASF GitHub Bot commented on YARN-5829:
--
Github user kambatla commented on a diff in the pull request:
https://github.com/apache/hadoop/pull/201#discussion_r108023073
--- Diff:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSSchedulerNode.java
---
@@ -110,16 +149,57 @@ synchronized FSAppAttempt getReservedAppSchedulable()
{
}
/**
+ * List reserved resources after preemption and assign them to the
+ * appropriate applications in a FIFO order.
+ * It is a good practice to call {@link #cleanupPreemptionList()}
+ * after this call
+ * @return if any resources were allocated
+ */
+ synchronized LinkedHashMap getPreemptionList() {
+cleanupPreemptionList();
+return new LinkedHashMap<>(resourcesPreemptedForApp);
+ }
+
+ /**
+ * Remove apps that have their preemption requests fulfilled
+ */
+ synchronized void cleanupPreemptionList() {
+Iterator iterator =
+resourcesPreemptedForApp.keySet().iterator();
+while (iterator.hasNext()) {
+ FSAppAttempt app = iterator.next();
+ boolean removeApp = false;
+ if (app.isStopped() || Resources.equals(
--- End diff --
Actually, we probably need to check if the app is starved instead of if it
having pending demand.
> FS preemption should reserve a node before considering containers on it for
> preemption
> --
>
> Key: YARN-5829
> URL: https://issues.apache.org/jira/browse/YARN-5829
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: fairscheduler
>Reporter: Karthik Kambatla
>Assignee: Miklos Szegedi
>
> FS preemption evaluates nodes for preemption, and subsequently preempts
> identified containers. If this node is not reserved for a specific
> application, any other application could be allocated resources on this node.
> Reserving the node for the starved application before preempting containers
> would help avoid this.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5829) FS preemption should reserve a node before considering containers on it for preemption
[
https://issues.apache.org/jira/browse/YARN-5829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15941419#comment-15941419
]
ASF GitHub Bot commented on YARN-5829:
--
Github user kambatla commented on a diff in the pull request:
https://github.com/apache/hadoop/pull/201#discussion_r108021321
--- Diff:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairScheduler.java
---
@@ -968,6 +966,32 @@ private boolean shouldContinueAssigning(int containers,
}
}
+ /**
+ * Assign preempted containers to the applications that have reserved
+ * resources for preempted containers.
+ * @param node Node to check
+ * @return assignment has occurred
+ */
+ static boolean assignPreemptedContainers(FSSchedulerNode node ) {
+boolean assignedAny = false;
+node.cleanupPreemptionList();
+for (Entry entry :
+node.getPreemptionList().entrySet()) {
+ FSAppAttempt app = entry.getKey();
+ Resource reserved = Resources.clone(entry.getValue());
--- End diff --
Can we call this variable pendingPreempted or some such. I would like us to
avoid calling this reserved :)
> FS preemption should reserve a node before considering containers on it for
> preemption
> --
>
> Key: YARN-5829
> URL: https://issues.apache.org/jira/browse/YARN-5829
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: fairscheduler
>Reporter: Karthik Kambatla
>Assignee: Miklos Szegedi
>
> FS preemption evaluates nodes for preemption, and subsequently preempts
> identified containers. If this node is not reserved for a specific
> application, any other application could be allocated resources on this node.
> Reserving the node for the starved application before preempting containers
> would help avoid this.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5829) FS preemption should reserve a node before considering containers on it for preemption
[
https://issues.apache.org/jira/browse/YARN-5829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15941426#comment-15941426
]
ASF GitHub Bot commented on YARN-5829:
--
Github user kambatla commented on a diff in the pull request:
https://github.com/apache/hadoop/pull/201#discussion_r108023542
--- Diff:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFSSchedulerNode.java
---
@@ -0,0 +1,376 @@
+package org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair;
+
+import org.apache.hadoop.yarn.api.records.*;
+import
org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainer;
+import org.apache.hadoop.yarn.server.resourcemanager.rmnode.RMNode;
+import org.apache.hadoop.yarn.util.resource.Resources;
+import org.junit.Test;
+import org.mockito.invocation.InvocationOnMock;
+import org.mockito.stubbing.Answer;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Map;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNotEquals;
+import static org.junit.Assert.assertTrue;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+/**
+ * Test scheduler node, especially preemption reservations.
+ */
+public class TestFSSchedulerNode {
+ private long containerNum = 0;
--- End diff --
Actually, can't you just use containers.size() instead?
> FS preemption should reserve a node before considering containers on it for
> preemption
> --
>
> Key: YARN-5829
> URL: https://issues.apache.org/jira/browse/YARN-5829
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: fairscheduler
>Reporter: Karthik Kambatla
>Assignee: Miklos Szegedi
>
> FS preemption evaluates nodes for preemption, and subsequently preempts
> identified containers. If this node is not reserved for a specific
> application, any other application could be allocated resources on this node.
> Reserving the node for the starved application before preempting containers
> would help avoid this.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Issue Comment Deleted] (YARN-5478) [YARN-4902] Define Java API for generalized & unified scheduling-strategies.
[ https://issues.apache.org/jira/browse/YARN-5478?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naganarasimha G R updated YARN-5478: Comment: was deleted (was: Hi [~wangda], Thanks for providing the patch ) > [YARN-4902] Define Java API for generalized & unified scheduling-strategies. > > > Key: YARN-5478 > URL: https://issues.apache.org/jira/browse/YARN-5478 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan >Assignee: Wangda Tan > Attachments: YARN-5478.1.patch, YARN-5478.2.patch, > YARN-5478.preliminary-poc.1.patch, YARN-5478.preliminary-poc.2.patch > > > Define Java API for application to specify generic scheduling requirements > described in YARN-4902 design doc. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5478) [YARN-4902] Define Java API for generalized & unified scheduling-strategies.
[ https://issues.apache.org/jira/browse/YARN-5478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15941403#comment-15941403 ] Naganarasimha G R commented on YARN-5478: - Hi [~wangda], Thanks for providing the patch > [YARN-4902] Define Java API for generalized & unified scheduling-strategies. > > > Key: YARN-5478 > URL: https://issues.apache.org/jira/browse/YARN-5478 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan >Assignee: Wangda Tan > Attachments: YARN-5478.1.patch, YARN-5478.2.patch, > YARN-5478.preliminary-poc.1.patch, YARN-5478.preliminary-poc.2.patch > > > Define Java API for application to specify generic scheduling requirements > described in YARN-4902 design doc. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6363) Extending SLS: Synthetic Load Generator
[ https://issues.apache.org/jira/browse/YARN-6363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15941390#comment-15941390 ] Carlo Curino commented on YARN-6363: [~wangda] in the last (v2) patch, I tried to make the SLSRunner to be a Tool, and fixed some of the shutdown logic, RM and SLSWebApp were not shutdown so I was getting port bind issues when I tried to make TestSLSRunner parametrized. Can you please check if that makes sense to you (I was in a rush)? Also I changed the commandline params to be more flexible on the tracetype (we should add an example rumen trace for TestSLSRunner) and I need to double check after the switch to Tool that this works. Are you ok with this new format? (using toolRunner we can probably do better with confs as welll). Also I think it would be nice to have a quick way to configure which ports to use (instead of a list in sls-runner.xml), because I think it would be nice to be able to run the SLSRunner in a map-only job where each mapper runs a different parametrization of the RM (ideal for parameter sweeps). Please share your thoughts. I will write up a bit about the configuration for the synth generator in the meantime. > Extending SLS: Synthetic Load Generator > --- > > Key: YARN-6363 > URL: https://issues.apache.org/jira/browse/YARN-6363 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Carlo Curino >Assignee: Carlo Curino > Attachments: YARN-6363.v0.patch, YARN-6363.v1.patch, > YARN-6363.v2.patch > > > This JIRA tracks the introduction of a synthetic load generator in the SLS. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6363) Extending SLS: Synthetic Load Generator
[ https://issues.apache.org/jira/browse/YARN-6363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Carlo Curino updated YARN-6363: --- Attachment: YARN-6363.v2.patch > Extending SLS: Synthetic Load Generator > --- > > Key: YARN-6363 > URL: https://issues.apache.org/jira/browse/YARN-6363 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Carlo Curino >Assignee: Carlo Curino > Attachments: YARN-6363.v0.patch, YARN-6363.v1.patch, > YARN-6363.v2.patch > > > This JIRA tracks the introduction of a synthetic load generator in the SLS. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6372) Add default value for NM disk validator
[
https://issues.apache.org/jira/browse/YARN-6372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15941366#comment-15941366
]
Hadoop QA commented on YARN-6372:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
11s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
3s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
49s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 9m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
34s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 13m
33s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
30s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 72m 31s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | YARN-6372 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12860470/YARN-6372.002.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux a1b43c143d1c 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20
09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 1f66524 |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/15384/testReport/ |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
U: hadoop-yarn-project/hadoop-yarn |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/15384/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message
[jira] [Commented] (YARN-5829) FS preemption should reserve a node before considering containers on it for preemption
[
https://issues.apache.org/jira/browse/YARN-5829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15941358#comment-15941358
]
ASF GitHub Bot commented on YARN-5829:
--
Github user kambatla commented on a diff in the pull request:
https://github.com/apache/hadoop/pull/201#discussion_r108020252
--- Diff:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSSchedulerNode.java
---
@@ -131,10 +203,58 @@ void
addContainersForPreemption(Collection containers) {
/**
* Remove container from the set of containers marked for preemption.
+ * Reserve the preempted resources for the app that requested
+ * the preemption.
*
* @param container container to remove
*/
void removeContainerForPreemption(RMContainer container) {
containersForPreemption.remove(container);
}
+
+ /**
+ * The Scheduler has allocated containers on this node to the given
+ * application.
+ * @param rmContainer Allocated container
+ * @param launchedOnNode True if the container has been launched
+ */
+ @Override
+ protected synchronized void allocateContainer(RMContainer rmContainer,
+boolean launchedOnNode) {
+super.allocateContainer(rmContainer, launchedOnNode);
+Resource allocated = rmContainer.getAllocatedResource();
+if (!Resources.isNone(allocated)) {
+ for (FSAppAttempt app: resourcesPreemptedPerApp.keySet()) {
--- End diff --
A for loop seems a little wasteful. Would it make any sense to keep another
map so this can just be a lookup?
> FS preemption should reserve a node before considering containers on it for
> preemption
> --
>
> Key: YARN-5829
> URL: https://issues.apache.org/jira/browse/YARN-5829
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: fairscheduler
>Reporter: Karthik Kambatla
>Assignee: Miklos Szegedi
>
> FS preemption evaluates nodes for preemption, and subsequently preempts
> identified containers. If this node is not reserved for a specific
> application, any other application could be allocated resources on this node.
> Reserving the node for the starved application before preempting containers
> would help avoid this.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6372) Add default value for NM disk validator
[ https://issues.apache.org/jira/browse/YARN-6372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15941355#comment-15941355 ] Miklos Szegedi commented on YARN-6372: -- +1 (non-binding) Thank you, [~yufeigu] for fixing this. > Add default value for NM disk validator > --- > > Key: YARN-6372 > URL: https://issues.apache.org/jira/browse/YARN-6372 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Affects Versions: 2.7.3, 3.0.0-alpha2 >Reporter: Yufei Gu >Assignee: Yufei Gu > Attachments: YARN-6372.001.patch, YARN-6372.002.patch > > > YARN-5137 make DiskChecker pluggable in NodeManager. We should give a default > value in case NM does't provide the configuration item. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6380) FSAppAttempt keeps redundant copy of the queue
[
https://issues.apache.org/jira/browse/YARN-6380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15941316#comment-15941316
]
Yufei Gu commented on YARN-6380:
Redundancy is evil. Thanks for fixing this. Would you mind fixing the casting
in getHeadRoom() since we are here?
{code}
public Resource getHeadroom() {
final FSQueue queue = (FSQueue) this.queue;
{code}
> FSAppAttempt keeps redundant copy of the queue
> --
>
> Key: YARN-6380
> URL: https://issues.apache.org/jira/browse/YARN-6380
> Project: Hadoop YARN
> Issue Type: Bug
> Components: fairscheduler
>Affects Versions: 3.0.0-alpha2
>Reporter: Daniel Templeton
>Assignee: Daniel Templeton
> Attachments: YARN-6380.001.patch, YARN-6380.002.patch
>
>
> The {{FSAppAttempt}} class defines its own {{fsQueue}} variable that is a
> second copy of the {{SchedulerApplicationAttempt}}'s {{queue}} variable.
> Aside from being redundant, it's also a bug, because when moving
> applications, we only update the {{SchedulerApplicationAttempt}}'s {{queue}},
> not the {{FSAppAttempt}}'s {{fsQueue}}.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5892) Capacity Scheduler: Support user-specific minimum user limit percent
[
https://issues.apache.org/jira/browse/YARN-5892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15941314#comment-15941314
]
Hadoop QA commented on YARN-5892:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
18s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
8s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 12m
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m
6s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
58s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
36s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
3s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
43s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 9m
26s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 58s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch
generated 16 new + 553 unchanged - 1 fixed = 569 total (was 554) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
23s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
27s{color} | {color:red}
hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager
generated 4 new + 882 unchanged - 0 fixed = 886 total (was 882) {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 40m
8s{color} | {color:green} hadoop-yarn-server-resourcemanager in the patch
passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
13s{color} | {color:green} hadoop-yarn-site in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
32s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 91m 41s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | YARN-5892 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12860457/YARN-5892.006.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 6935e8b416fc 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6
15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision |
[jira] [Commented] (YARN-2113) Add cross-user preemption within CapacityScheduler's leaf-queue
[
https://issues.apache.org/jira/browse/YARN-2113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15941308#comment-15941308
]
Wangda Tan commented on YARN-2113:
--
[~sunilg], thanks for update the patch.
Checked the patch and also offline discussed with Sunil, I can understand that
this patch handles an edge case which allows a lower priority app preempt
higher priority app if the user of higher priority app beyond its user limit.
However I think it is debatable: as of now, YARN doesn't calculate UL for
different priorities, but that doesn't mean we should allow lower priority app
to preempt higher priority app. [~eepayne], could you share your thoughts here?
And also, this patch added a compound comparator which sort application in
following way (essentially): {{(priority > demand) > (fifo > demand)}}
I'm a little worried about this complex comparator. If we could decide no lower
priority app can preempt higher priority app. We should be able to simply
compare logic to:
{code}
compare(a, b)
if (a.priority != b.priority) {
return a.priority.compare(b.priority)
}
return a.toBePreemptFromOther.compare(b.toBePreemptFromOther)
{code}
Above logic makes sure that when priority is same, app with
toBePreemptFromOther always goes first. Is it possible that one app's
toBePreemptFromOther > 0 and actuallyToBePreempted > 0 at the same time? If it
is possible, could you add some examples?
> Add cross-user preemption within CapacityScheduler's leaf-queue
> ---
>
> Key: YARN-2113
> URL: https://issues.apache.org/jira/browse/YARN-2113
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: scheduler
>Reporter: Vinod Kumar Vavilapalli
>Assignee: Sunil G
> Attachments: YARN-2113.0001.patch, YARN-2113.0002.patch,
> YARN-2113.0003.patch, YARN-2113.v0.patch
>
>
> Preemption today only works across queues and moves around resources across
> queues per demand and usage. We should also have user-level preemption within
> a queue, to balance capacity across users in a predictable manner.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6372) Add default value for NM disk validator
[ https://issues.apache.org/jira/browse/YARN-6372?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yufei Gu updated YARN-6372: --- Attachment: YARN-6372.002.patch Uploaded patch v2 for Miklos' comment. > Add default value for NM disk validator > --- > > Key: YARN-6372 > URL: https://issues.apache.org/jira/browse/YARN-6372 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Affects Versions: 2.7.3, 3.0.0-alpha2 >Reporter: Yufei Gu >Assignee: Yufei Gu > Attachments: YARN-6372.001.patch, YARN-6372.002.patch > > > YARN-5137 make DiskChecker pluggable in NodeManager. We should give a default > value in case NM does't provide the configuration item. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-3427) Remove deprecated methods from ResourceCalculatorProcessTree
[ https://issues.apache.org/jira/browse/YARN-3427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15941291#comment-15941291 ] Daniel Templeton commented on YARN-3427: LGTM +1 I'll commit it next week. CC: [~hitesh] > Remove deprecated methods from ResourceCalculatorProcessTree > > > Key: YARN-3427 > URL: https://issues.apache.org/jira/browse/YARN-3427 > Project: Hadoop YARN > Issue Type: Improvement >Affects Versions: 2.7.0 >Reporter: Karthik Kambatla >Assignee: Miklos Szegedi >Priority: Blocker > Attachments: YARN-3427.000.patch, YARN-3427.001.patch > > > In 2.7, we made ResourceCalculatorProcessTree Public and exposed some > existing ill-formed methods as deprecated ones for use by Tez. > We should remove it in 3.0.0, considering that the methods have been > deprecated for the all 2.x.y releases that it is marked Public in. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-3427) Remove deprecated methods from ResourceCalculatorProcessTree
[
https://issues.apache.org/jira/browse/YARN-3427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15941286#comment-15941286
]
Hadoop QA commented on YARN-3427:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
35s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
57s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 19s{color} | {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common: The patch generated 1 new +
126 unchanged - 21 fixed = 127 total (was 147) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
21s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 24m 28s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | YARN-3427 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12860462/YARN-3427.001.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 76dac8689952 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6
15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 1f66524 |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/15383/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/15383/testReport/ |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/15383/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Remove deprecated methods from ResourceCalculatorProcessTree
>
>
> Key: YARN-3427
> URL: https://issues.apache.org/jira/browse/YARN-3427
> Project: Hadoop YARN
> Issue Type: Improvement
>Affects
[jira] [Updated] (YARN-6108) Improve AHS webservice to accept NM address as a parameter to get container logs
[ https://issues.apache.org/jira/browse/YARN-6108?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated YARN-6108: -- Fix Version/s: (was: 3.0.0-beta1) 3.0.0-alpha3 > Improve AHS webservice to accept NM address as a parameter to get container > logs > > > Key: YARN-6108 > URL: https://issues.apache.org/jira/browse/YARN-6108 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Xuan Gong >Assignee: Xuan Gong > Fix For: 2.9.0, 3.0.0-alpha3 > > Attachments: YARN-6108.1.patch, YARN-6108.2.patch, YARN-6108.3.patch, > YARN-6108.4.patch, YARN-6108.5.patch, YARN-6108.5.rebase.patch, > YARN-6108.branch-2.v1.patch > > > Currently, if we want to get container log for running application, we need > to get NM web address from AHS which we need to enable > yarn.timeline-service.generic-application-history.save-non-am-container-meta-info > for non-am containers. But, in most of time, we will disable this > configuration for ATS performance purpose. In this case, it is impossible for > us to get the logs for non-am container in a running application. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-3427) Remove deprecated methods from ResourceCalculatorProcessTree
[ https://issues.apache.org/jira/browse/YARN-3427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Szegedi updated YARN-3427: - Attachment: YARN-3427.001.patch Good point. I updated the patch. > Remove deprecated methods from ResourceCalculatorProcessTree > > > Key: YARN-3427 > URL: https://issues.apache.org/jira/browse/YARN-3427 > Project: Hadoop YARN > Issue Type: Improvement >Affects Versions: 2.7.0 >Reporter: Karthik Kambatla >Assignee: Miklos Szegedi >Priority: Blocker > Attachments: YARN-3427.000.patch, YARN-3427.001.patch > > > In 2.7, we made ResourceCalculatorProcessTree Public and exposed some > existing ill-formed methods as deprecated ones for use by Tez. > We should remove it in 3.0.0, considering that the methods have been > deprecated for the all 2.x.y releases that it is marked Public in. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6357) Implement putEntitiesAsync API in TimelineCollector
[
https://issues.apache.org/jira/browse/YARN-6357?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15941236#comment-15941236
]
Varun Saxena commented on YARN-6357:
Thanks Haibo. Will commit it in a day.
> Implement putEntitiesAsync API in TimelineCollector
> ---
>
> Key: YARN-6357
> URL: https://issues.apache.org/jira/browse/YARN-6357
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: ATSv2, timelineserver
>Affects Versions: YARN-2928
>Reporter: Joep Rottinghuis
>Assignee: Haibo Chen
> Labels: yarn-5355-merge-blocker
> Attachments: YARN-6357.01.patch, YARN-6357.02.patch,
> YARN-6357.03.patch, YARN-6357.04.patch
>
>
> As discovered and discussed in YARN-5269 the
> TimelineCollector#putEntitiesAsync method is currently not implemented and
> TimelineCollector#putEntities is asynchronous.
> TimelineV2ClientImpl#putEntities vs TimelineV2ClientImpl#putEntitiesAsync
> correctly call TimelineEntityDispatcher#dispatchEntities(boolean sync,...
> with the correct argument. This argument does seem to make it into the
> params, and on the server side TimelineCollectorWebService#putEntities
> correctly pulls the async parameter from the rest call. See line 156:
> {code}
> boolean isAsync = async != null && async.trim().equalsIgnoreCase("true");
> {code}
> However, this is where the problem starts. It simply calls
> TimelineCollector#putEntities and ignores the value of isAsync. It should
> instead have called TimelineCollector#putEntitiesAsync, which is currently
> not implemented.
> putEntities should call putEntitiesAsync and then after that call
> writer.flush()
> The fact that we flush on close and we flush periodically should be more of a
> concern of avoiding data loss; close in case sync is never called and the
> periodic flush to guard against having data from slow writers get buffered
> for a long time and expose us to risk of loss in case the collector crashes
> with data in its buffers. Size-based flush is a different concern to avoid
> blowing up memory footprint.
> The spooling behavior is also somewhat separate.
> We have two separate methods on our API putEntities and putEntitiesAsync and
> they should have different behavior beyond waiting for the request to be sent.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5892) Capacity Scheduler: Support user-specific minimum user limit percent
[
https://issues.apache.org/jira/browse/YARN-5892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Payne updated YARN-5892:
-
Attachment: YARN-5892.006.patch
Thanks for your helpful comments [~leftnoteasy] and [~sunilg]. I am attaching a
new patch where the user limit computations are done as before but then the
weight is applied afterwards. It is simpler.
Thanks.
> Capacity Scheduler: Support user-specific minimum user limit percent
>
>
> Key: YARN-5892
> URL: https://issues.apache.org/jira/browse/YARN-5892
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: capacityscheduler
>Reporter: Eric Payne
>Assignee: Eric Payne
> Attachments: Active users highlighted.jpg, YARN-5892.001.patch,
> YARN-5892.002.patch, YARN-5892.003.patch, YARN-5892.004.patch,
> YARN-5892.005.patch, YARN-5892.006.patch
>
>
> Currently, in the capacity scheduler, the {{minimum-user-limit-percent}}
> property is per queue. A cluster admin should be able to set the minimum user
> limit percent on a per-user basis within the queue.
> This functionality is needed so that when intra-queue preemption is enabled
> (YARN-4945 / YARN-2113), some users can be deemed as more important than
> other users, and resources from VIP users won't be as likely to be preempted.
> For example, if the {{getstuffdone}} queue has a MULP of 25 percent, but user
> {{jane}} is a power user of queue {{getstuffdone}} and needs to be guaranteed
> 75 percent, the properties for {{getstuffdone}} and {{jane}} would look like
> this:
> {code}
>
>
> yarn.scheduler.capacity.root.getstuffdone.minimum-user-limit-percent
> 25
>
>
>
> yarn.scheduler.capacity.root.getstuffdone.jane.minimum-user-limit-percent
> 75
>
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5685) Non-embedded HA failover is broken
[
https://issues.apache.org/jira/browse/YARN-5685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Templeton updated YARN-5685:
---
Attachment: YARN-5685.005.patch
Updated patch for [~kasha]'s comments.
> Non-embedded HA failover is broken
> --
>
> Key: YARN-5685
> URL: https://issues.apache.org/jira/browse/YARN-5685
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
>Affects Versions: 2.9.0, 3.0.0-alpha1
>Reporter: Daniel Templeton
>Assignee: Daniel Templeton
>Priority: Critical
> Labels: oct16-hard
> Attachments: YARN-5685.001.patch, YARN-5685.002.patch,
> YARN-5685.003.patch, YARN-5685.004.patch, YARN-5685.005.patch
>
>
> If HA is enabled with automatic failover enabled and embedded failover
> disabled, all RMs all come up in standby state. To make one of them active,
> the {{\-\-forcemanual}} flag must be used when manually triggering the state
> change. Should the active go down, the standby will not become active and
> must be manually transitioned with the {{\-\-forcemanual}} flag.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-6359) TestRM#testApplicationKillAtAcceptedState fails rarely due to race condition
[
https://issues.apache.org/jira/browse/YARN-6359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15941129#comment-15941129
]
Karthik Kambatla edited comment on YARN-6359 at 3/24/17 9:04 PM:
-
Nit: I think we use 100 ms retry in other places; maybe we should use the same?
was (Author: kasha):
Nit: I think we use 100 ms in other places; maybe we should use the same?
> TestRM#testApplicationKillAtAcceptedState fails rarely due to race condition
>
>
> Key: YARN-6359
> URL: https://issues.apache.org/jira/browse/YARN-6359
> Project: Hadoop YARN
> Issue Type: Bug
> Components: test
>Affects Versions: 2.9.0, 3.0.0-alpha3
>Reporter: Robert Kanter
>Assignee: Robert Kanter
> Attachments: YARN-6359.001.patch, YARN-6359.002.patch
>
>
> We've seen (very rarely) a test failure in
> {{TestRM#testApplicationKillAtAcceptedState}}
> {noformat}
> java.lang.AssertionError: expected:<1> but was:<0>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at org.junit.Assert.assertEquals(Assert.java:542)
> at
> org.apache.hadoop.yarn.server.resourcemanager.TestRM.testApplicationKillAtAcceptedState(TestRM.java:645)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6359) TestRM#testApplicationKillAtAcceptedState fails rarely due to race condition
[
https://issues.apache.org/jira/browse/YARN-6359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15941129#comment-15941129
]
Karthik Kambatla commented on YARN-6359:
Nit: I think we use 100 ms in other places; maybe we should use the same?
> TestRM#testApplicationKillAtAcceptedState fails rarely due to race condition
>
>
> Key: YARN-6359
> URL: https://issues.apache.org/jira/browse/YARN-6359
> Project: Hadoop YARN
> Issue Type: Bug
> Components: test
>Affects Versions: 2.9.0, 3.0.0-alpha3
>Reporter: Robert Kanter
>Assignee: Robert Kanter
> Attachments: YARN-6359.001.patch, YARN-6359.002.patch
>
>
> We've seen (very rarely) a test failure in
> {{TestRM#testApplicationKillAtAcceptedState}}
> {noformat}
> java.lang.AssertionError: expected:<1> but was:<0>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at org.junit.Assert.assertEquals(Assert.java:542)
> at
> org.apache.hadoop.yarn.server.resourcemanager.TestRM.testApplicationKillAtAcceptedState(TestRM.java:645)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6334) TestRMFailover#testAutomaticFailover always passes even when it should fail
[
https://issues.apache.org/jira/browse/YARN-6334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15941127#comment-15941127
]
Daniel Templeton commented on YARN-6334:
Committed to branch-2. Thanks, [~yufeigu].
> TestRMFailover#testAutomaticFailover always passes even when it should fail
> ---
>
> Key: YARN-6334
> URL: https://issues.apache.org/jira/browse/YARN-6334
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Yufei Gu
>Assignee: Yufei Gu
> Fix For: 2.9.0, 3.0.0-alpha3
>
> Attachments: YARN-6334.001.patch, YARN-6334.002.patch,
> YARN-6334.003.patch, YARN-6334.004.patch, YARN-6334.005.patch,
> YARN-6334.branch2.001.patch
>
>
> Due to a bug in {{while}} loop.
> {code}
> int maxWaitingAttempts = 2000;
> while (maxWaitingAttempts-- > 0 ) {
> if (rm.getRMContext().getHAServiceState() == HAServiceState.STANDBY) {
> break;
> }
> Thread.sleep(1);
> }
> Assert.assertFalse("RM didn't transition to Standby ",
> maxWaitingAttempts == 0);
> {code}
> maxWaitingAttempts is -1 if RM didn't transition to Standby.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6334) TestRMFailover#testAutomaticFailover always passes even when it should fail
[
https://issues.apache.org/jira/browse/YARN-6334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Templeton updated YARN-6334:
---
Fix Version/s: 2.9.0
> TestRMFailover#testAutomaticFailover always passes even when it should fail
> ---
>
> Key: YARN-6334
> URL: https://issues.apache.org/jira/browse/YARN-6334
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Yufei Gu
>Assignee: Yufei Gu
> Fix For: 2.9.0, 3.0.0-alpha3
>
> Attachments: YARN-6334.001.patch, YARN-6334.002.patch,
> YARN-6334.003.patch, YARN-6334.004.patch, YARN-6334.005.patch,
> YARN-6334.branch2.001.patch
>
>
> Due to a bug in {{while}} loop.
> {code}
> int maxWaitingAttempts = 2000;
> while (maxWaitingAttempts-- > 0 ) {
> if (rm.getRMContext().getHAServiceState() == HAServiceState.STANDBY) {
> break;
> }
> Thread.sleep(1);
> }
> Assert.assertFalse("RM didn't transition to Standby ",
> maxWaitingAttempts == 0);
> {code}
> maxWaitingAttempts is -1 if RM didn't transition to Standby.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-2113) Add cross-user preemption within CapacityScheduler's leaf-queue
[
https://issues.apache.org/jira/browse/YARN-2113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15941124#comment-15941124
]
Hadoop QA commented on YARN-2113:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
23s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
16s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
58s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 23s{color} | {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 23 new + 92 unchanged - 2 fixed = 115 total (was 94)
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 40m 14s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 61m 57s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.yarn.server.resourcemanager.TestRMRestart |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | YARN-2113 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12860432/YARN-2113.0003.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 706b4b0301c1 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6
15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 52b0060 |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/15380/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
|
| unit |
https://builds.apache.org/job/PreCommit-YARN-Build/15380/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/15380/testReport/ |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/15380/console
[jira] [Commented] (YARN-5685) Non-embedded HA failover is broken
[
https://issues.apache.org/jira/browse/YARN-5685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15941122#comment-15941122
]
Karthik Kambatla commented on YARN-5685:
Have no recollection from my previous review. Looked through again. Minor
comments:
# In the javadoc for {{verifyLeaderElection}}, the @link annotation is missing
the closing brace.
# In TestRMAdminService, can we split the three cases in the test to three
individual tests.
> Non-embedded HA failover is broken
> --
>
> Key: YARN-5685
> URL: https://issues.apache.org/jira/browse/YARN-5685
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
>Affects Versions: 2.9.0, 3.0.0-alpha1
>Reporter: Daniel Templeton
>Assignee: Daniel Templeton
>Priority: Critical
> Labels: oct16-hard
> Attachments: YARN-5685.001.patch, YARN-5685.002.patch,
> YARN-5685.003.patch, YARN-5685.004.patch
>
>
> If HA is enabled with automatic failover enabled and embedded failover
> disabled, all RMs all come up in standby state. To make one of them active,
> the {{\-\-forcemanual}} flag must be used when manually triggering the state
> change. Should the active go down, the standby will not become active and
> must be manually transitioned with the {{\-\-forcemanual}} flag.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6357) Implement putEntitiesAsync API in TimelineCollector
[
https://issues.apache.org/jira/browse/YARN-6357?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15941120#comment-15941120
]
Hadoop QA commented on YARN-6357:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
16s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
39s{color} | {color:green} hadoop-yarn-server-timelineservice in the patch
passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 19m 37s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | YARN-6357 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12860442/YARN-6357.04.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 27d4f4182342 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6
15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 332a997 |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/15381/testReport/ |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/15381/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Implement putEntitiesAsync API in TimelineCollector
> ---
>
> Key: YARN-6357
> URL: https://issues.apache.org/jira/browse/YARN-6357
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: ATSv2, timelineserver
>Affects Versions: YARN-2928
>Reporter: Joep Rottinghuis
>Assignee: Haibo Chen
> Labels: yarn-5355-merge-blocker
> Attachments:
[jira] [Updated] (YARN-6334) TestRMFailover#testAutomaticFailover always passes even when it should fail
[
https://issues.apache.org/jira/browse/YARN-6334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yufei Gu updated YARN-6334:
---
Attachment: YARN-6334.branch2.001.patch
Uploaded branch 2 version. Thanks [~templedf] for the review and commit. Thanks
[~haibochen] for the review.
> TestRMFailover#testAutomaticFailover always passes even when it should fail
> ---
>
> Key: YARN-6334
> URL: https://issues.apache.org/jira/browse/YARN-6334
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Yufei Gu
>Assignee: Yufei Gu
> Fix For: 3.0.0-alpha3
>
> Attachments: YARN-6334.001.patch, YARN-6334.002.patch,
> YARN-6334.003.patch, YARN-6334.004.patch, YARN-6334.005.patch,
> YARN-6334.branch2.001.patch
>
>
> Due to a bug in {{while}} loop.
> {code}
> int maxWaitingAttempts = 2000;
> while (maxWaitingAttempts-- > 0 ) {
> if (rm.getRMContext().getHAServiceState() == HAServiceState.STANDBY) {
> break;
> }
> Thread.sleep(1);
> }
> Assert.assertFalse("RM didn't transition to Standby ",
> maxWaitingAttempts == 0);
> {code}
> maxWaitingAttempts is -1 if RM didn't transition to Standby.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6357) Implement putEntitiesAsync API in TimelineCollector
[
https://issues.apache.org/jira/browse/YARN-6357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haibo Chen updated YARN-6357:
-
Attachment: YARN-6357.04.patch
Updated the patch according to your catch.
> Implement putEntitiesAsync API in TimelineCollector
> ---
>
> Key: YARN-6357
> URL: https://issues.apache.org/jira/browse/YARN-6357
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: ATSv2, timelineserver
>Affects Versions: YARN-2928
>Reporter: Joep Rottinghuis
>Assignee: Haibo Chen
> Labels: yarn-5355-merge-blocker
> Attachments: YARN-6357.01.patch, YARN-6357.02.patch,
> YARN-6357.03.patch, YARN-6357.04.patch
>
>
> As discovered and discussed in YARN-5269 the
> TimelineCollector#putEntitiesAsync method is currently not implemented and
> TimelineCollector#putEntities is asynchronous.
> TimelineV2ClientImpl#putEntities vs TimelineV2ClientImpl#putEntitiesAsync
> correctly call TimelineEntityDispatcher#dispatchEntities(boolean sync,...
> with the correct argument. This argument does seem to make it into the
> params, and on the server side TimelineCollectorWebService#putEntities
> correctly pulls the async parameter from the rest call. See line 156:
> {code}
> boolean isAsync = async != null && async.trim().equalsIgnoreCase("true");
> {code}
> However, this is where the problem starts. It simply calls
> TimelineCollector#putEntities and ignores the value of isAsync. It should
> instead have called TimelineCollector#putEntitiesAsync, which is currently
> not implemented.
> putEntities should call putEntitiesAsync and then after that call
> writer.flush()
> The fact that we flush on close and we flush periodically should be more of a
> concern of avoiding data loss; close in case sync is never called and the
> periodic flush to guard against having data from slow writers get buffered
> for a long time and expose us to risk of loss in case the collector crashes
> with data in its buffers. Size-based flush is a different concern to avoid
> blowing up memory footprint.
> The spooling behavior is also somewhat separate.
> We have two separate methods on our API putEntities and putEntitiesAsync and
> they should have different behavior beyond waiting for the request to be sent.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6339) Improve performance for createAndGetApplicationReport
[ https://issues.apache.org/jira/browse/YARN-6339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15941071#comment-15941071 ] Xuan Gong commented on YARN-6339: - [~zhaoyunjiong] Thanks for the fix. +1 LGTM > Improve performance for createAndGetApplicationReport > - > > Key: YARN-6339 > URL: https://issues.apache.org/jira/browse/YARN-6339 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: yunjiong zhao >Assignee: yunjiong zhao > Attachments: YARN-6339.001.patch, YARN-6339.002.patch, > YARN-6339.003.patch > > > There are two performance issue when calling createAndGetApplicationReport: > One is inside ProtoUtils.convertFromProtoFormat, replace is too slow for > clusters which have more than 3000 nodes. Use substring is much better: > https://issues.apache.org/jira/browse/YARN-6285?focusedCommentId=15923241=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15923241 > Another one is inside getLogAggregationReportsForApp, if some application's > LogAggregationStatus is TIME_OUT, every time it was called it will create an > HashMap which will produce lots of garbage. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5685) Non-embedded HA failover is broken
[
https://issues.apache.org/jira/browse/YARN-5685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Templeton updated YARN-5685:
---
Attachment: YARN-5685.004.patch
New patch to address [~kasha]'s comments.
> Non-embedded HA failover is broken
> --
>
> Key: YARN-5685
> URL: https://issues.apache.org/jira/browse/YARN-5685
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
>Affects Versions: 2.9.0, 3.0.0-alpha1
>Reporter: Daniel Templeton
>Assignee: Daniel Templeton
>Priority: Critical
> Labels: oct16-hard
> Attachments: YARN-5685.001.patch, YARN-5685.002.patch,
> YARN-5685.003.patch, YARN-5685.004.patch
>
>
> If HA is enabled with automatic failover enabled and embedded failover
> disabled, all RMs all come up in standby state. To make one of them active,
> the {{\-\-forcemanual}} flag must be used when manually triggering the state
> change. Should the active go down, the standby will not become active and
> must be manually transitioned with the {{\-\-forcemanual}} flag.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6302) Fail the node, if Linux Container Executor is not configured properly
[
https://issues.apache.org/jira/browse/YARN-6302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15941039#comment-15941039
]
ASF GitHub Bot commented on YARN-6302:
--
Github user templedf commented on a diff in the pull request:
https://github.com/apache/hadoop/pull/200#discussion_r107987190
--- Diff:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/LinuxContainerExecutor.java
---
@@ -551,16 +550,16 @@ public int launchContainer(ContainerStartContext ctx)
} else {
LOG.info(
"Container was marked as inactive. Returning terminated
error");
-return ExitCode.TERMINATED.getExitCode();
+return ContainerExecutor.ExitCode.TERMINATED.getExitCode();
--- End diff --
I don't think this is needful, but you can do it if you want.
> Fail the node, if Linux Container Executor is not configured properly
> -
>
> Key: YARN-6302
> URL: https://issues.apache.org/jira/browse/YARN-6302
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Miklos Szegedi
>Assignee: Miklos Szegedi
>Priority: Minor
>
> We have a cluster that has one node with misconfigured Linux Container
> Executor. Every time an AM or regular container is launched on the cluster,
> it will fail. The node will still have resources available, so it keeps
> failing apps until the administrator notices the issue and decommissions the
> node. AM Blacklisting only helps, if the application is already running.
> As a possible improvement, when the LCE is used on the cluster and a NM gets
> certain errors back from the LCE, like error 24 configuration not found, we
> should not try to allocate anything on the node anymore or shut down the node
> entirely. That kind of problem normally does not fix itself and it means that
> nothing can really run on that node.
> {code}
> Application application_1488920587909_0010 failed 2 times due to AM Container
> for appattempt_1488920587909_0010_02 exited with exitCode: -1000
> Failing this attempt.Diagnostics: Application application_1488920587909_0010
> initialization failed (exitCode=24) with output:
> For more detailed output, check the application tracking page:
> http://node-1.domain.com:8088/cluster/app/application_1488920587909_0010 Then
> click on links to logs of each attempt.
> . Failing the application.
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6302) Fail the node, if Linux Container Executor is not configured properly
[
https://issues.apache.org/jira/browse/YARN-6302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15941040#comment-15941040
]
ASF GitHub Bot commented on YARN-6302:
--
Github user templedf commented on a diff in the pull request:
https://github.com/apache/hadoop/pull/200#discussion_r107987010
--- Diff:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/LinuxContainerExecutor.java
---
@@ -580,19 +579,19 @@ public int launchContainer(ContainerStartContext ctx)
logOutput(diagnostics);
container.handle(new ContainerDiagnosticsUpdateEvent(containerId,
diagnostics));
-if (exitCode == LinuxContainerExecutorExitCode.
+if (exitCode == ExitCode.
--- End diff --
Sorry to pick, but can we split these lines on the == instead of the . ?
> Fail the node, if Linux Container Executor is not configured properly
> -
>
> Key: YARN-6302
> URL: https://issues.apache.org/jira/browse/YARN-6302
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Miklos Szegedi
>Assignee: Miklos Szegedi
>Priority: Minor
>
> We have a cluster that has one node with misconfigured Linux Container
> Executor. Every time an AM or regular container is launched on the cluster,
> it will fail. The node will still have resources available, so it keeps
> failing apps until the administrator notices the issue and decommissions the
> node. AM Blacklisting only helps, if the application is already running.
> As a possible improvement, when the LCE is used on the cluster and a NM gets
> certain errors back from the LCE, like error 24 configuration not found, we
> should not try to allocate anything on the node anymore or shut down the node
> entirely. That kind of problem normally does not fix itself and it means that
> nothing can really run on that node.
> {code}
> Application application_1488920587909_0010 failed 2 times due to AM Container
> for appattempt_1488920587909_0010_02 exited with exitCode: -1000
> Failing this attempt.Diagnostics: Application application_1488920587909_0010
> initialization failed (exitCode=24) with output:
> For more detailed output, check the application tracking page:
> http://node-1.domain.com:8088/cluster/app/application_1488920587909_0010 Then
> click on links to logs of each attempt.
> . Failing the application.
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-2113) Add cross-user preemption within CapacityScheduler's leaf-queue
[ https://issues.apache.org/jira/browse/YARN-2113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil G updated YARN-2113: -- Attachment: YARN-2113.0003.patch handles all test failures and added one more test case to cover a user-limit + priority test case. test cases are passing locally. > Add cross-user preemption within CapacityScheduler's leaf-queue > --- > > Key: YARN-2113 > URL: https://issues.apache.org/jira/browse/YARN-2113 > Project: Hadoop YARN > Issue Type: Sub-task > Components: scheduler >Reporter: Vinod Kumar Vavilapalli >Assignee: Sunil G > Attachments: YARN-2113.0001.patch, YARN-2113.0002.patch, > YARN-2113.0003.patch, YARN-2113.v0.patch > > > Preemption today only works across queues and moves around resources across > queues per demand and usage. We should also have user-level preemption within > a queue, to balance capacity across users in a predictable manner. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Resolved] (YARN-6388) Add YARN_TIMELINEREADER_OPTS into yarn-env.sh
[ https://issues.apache.org/jira/browse/YARN-6388?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rohith Sharma K S resolved YARN-6388. - Resolution: Duplicate > Add YARN_TIMELINEREADER_OPTS into yarn-env.sh > - > > Key: YARN-6388 > URL: https://issues.apache.org/jira/browse/YARN-6388 > Project: Hadoop YARN > Issue Type: Bug > Components: timelinereader >Reporter: Rohith Sharma K S > Labels: yarn-5355-merge-blocker > > yarn-env.sh does not contains YARN_TIMELINEREADER_OPTS. This need to be added > into yarn-env.sh so that process specific opts can be passed into timeline > reader. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6388) Add YARN_TIMELINEREADER_OPTS into yarn-env.sh
[ https://issues.apache.org/jira/browse/YARN-6388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15940994#comment-15940994 ] Rohith Sharma K S commented on YARN-6388: - My bad! Since it is not fixed in YARN-5355 branch, I thought it is not added. Thanks for the pointer. I will close it as duplicate. > Add YARN_TIMELINEREADER_OPTS into yarn-env.sh > - > > Key: YARN-6388 > URL: https://issues.apache.org/jira/browse/YARN-6388 > Project: Hadoop YARN > Issue Type: Bug > Components: timelinereader >Reporter: Rohith Sharma K S > Labels: yarn-5355-merge-blocker > > yarn-env.sh does not contains YARN_TIMELINEREADER_OPTS. This need to be added > into yarn-env.sh so that process specific opts can be passed into timeline > reader. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6334) TestRMFailover#testAutomaticFailover always passes even when it should fail
[
https://issues.apache.org/jira/browse/YARN-6334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15940984#comment-15940984
]
Hudson commented on YARN-6334:
--
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11456 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/11456/])
YARN-6334. TestRMFailover#testAutomaticFailover always passes even when
(templedf: rev 33815af4242ac8c6b119128730e63f13164fd763)
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/TestRMFailover.java
> TestRMFailover#testAutomaticFailover always passes even when it should fail
> ---
>
> Key: YARN-6334
> URL: https://issues.apache.org/jira/browse/YARN-6334
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Yufei Gu
>Assignee: Yufei Gu
> Fix For: 3.0.0-alpha3
>
> Attachments: YARN-6334.001.patch, YARN-6334.002.patch,
> YARN-6334.003.patch, YARN-6334.004.patch, YARN-6334.005.patch
>
>
> Due to a bug in {{while}} loop.
> {code}
> int maxWaitingAttempts = 2000;
> while (maxWaitingAttempts-- > 0 ) {
> if (rm.getRMContext().getHAServiceState() == HAServiceState.STANDBY) {
> break;
> }
> Thread.sleep(1);
> }
> Assert.assertFalse("RM didn't transition to Standby ",
> maxWaitingAttempts == 0);
> {code}
> maxWaitingAttempts is -1 if RM didn't transition to Standby.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-6389) [ATSv2] metricslimit query parameter do not work
[
https://issues.apache.org/jira/browse/YARN-6389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15940975#comment-15940975
]
Rohith Sharma K S edited comment on YARN-6389 at 3/24/17 7:20 PM:
--
Btw, I am using hbase-1.1.5. But the same hbase version is used in old build
where metricslimit is working fine.
And also I looked into tests modifying a bit. It appears in test
{{metricslimit}} is working fine for ApplicationEntity. So may be issue with
GenericEntity publishing.
was (Author: rohithsharma):
Btw, I am using hbase-1.1.5. But the same hbase version is used in old build
where metricslimit is working fine.
And also I looked into tests modifying a bit. But it appears in test
{{metricslimit}} is working fine.
> [ATSv2] metricslimit query parameter do not work
>
>
> Key: YARN-6389
> URL: https://issues.apache.org/jira/browse/YARN-6389
> Project: Hadoop YARN
> Issue Type: Bug
> Components: timelinereader
>Reporter: Rohith Sharma K S
>Assignee: Varun Saxena
>Priority: Critical
>
> It is observed that metricslimit query parameter do not work. And also by
> default metricslimit is set to 1, all the metrics versions are retrieved.
> One thing I noticed that even though GenericEntityReader is setting
> Scan.setMaxVersions(), all the metrics are retrieved. It appears something
> wrong in TimelieneWritter or the way hbase filters is being used.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6389) [ATSv2] metricslimit query parameter do not work
[
https://issues.apache.org/jira/browse/YARN-6389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15940975#comment-15940975
]
Rohith Sharma K S commented on YARN-6389:
-
Btw, I am using hbase-1.1.5. But the same hbase version is used in old build
where metricslimit is working fine.
And also I looked into tests modifying a bit. But it appears in test
{{metricslimit}} is working fine.
> [ATSv2] metricslimit query parameter do not work
>
>
> Key: YARN-6389
> URL: https://issues.apache.org/jira/browse/YARN-6389
> Project: Hadoop YARN
> Issue Type: Bug
> Components: timelinereader
>Reporter: Rohith Sharma K S
>Assignee: Varun Saxena
>Priority: Critical
>
> It is observed that metricslimit query parameter do not work. And also by
> default metricslimit is set to 1, all the metrics versions are retrieved.
> One thing I noticed that even though GenericEntityReader is setting
> Scan.setMaxVersions(), all the metrics are retrieved. It appears something
> wrong in TimelieneWritter or the way hbase filters is being used.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-3427) Remove deprecated methods from ResourceCalculatorProcessTree
[
https://issues.apache.org/jira/browse/YARN-3427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15940967#comment-15940967
]
Daniel Templeton commented on YARN-3427:
Is it now safe to also remove the suppression on the
{{testMemForOlderProcesses()}} method?
> Remove deprecated methods from ResourceCalculatorProcessTree
>
>
> Key: YARN-3427
> URL: https://issues.apache.org/jira/browse/YARN-3427
> Project: Hadoop YARN
> Issue Type: Improvement
>Affects Versions: 2.7.0
>Reporter: Karthik Kambatla
>Assignee: Miklos Szegedi
>Priority: Blocker
> Attachments: YARN-3427.000.patch
>
>
> In 2.7, we made ResourceCalculatorProcessTree Public and exposed some
> existing ill-formed methods as deprecated ones for use by Tez.
> We should remove it in 3.0.0, considering that the methods have been
> deprecated for the all 2.x.y releases that it is marked Public in.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5654) Not be able to run SLS with FairScheduler
[
https://issues.apache.org/jira/browse/YARN-5654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15940962#comment-15940962
]
Hadoop QA commented on YARN-5654:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
21s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
16s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
16s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
19s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 11s{color} | {color:orange} hadoop-tools/hadoop-sls: The patch generated 5
new + 94 unchanged - 133 fixed = 99 total (was 227) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
53s{color} | {color:green} hadoop-sls in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 19m 44s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | YARN-5654 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12860423/YARN-5654.003.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 86815f289c28 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20
09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 52b0060 |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/15379/artifact/patchprocess/diff-checkstyle-hadoop-tools_hadoop-sls.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/15379/testReport/ |
| modules | C: hadoop-tools/hadoop-sls U: hadoop-tools/hadoop-sls |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/15379/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Not be able to run SLS with FairScheduler
> -
>
> Key: YARN-5654
> URL: https://issues.apache.org/jira/browse/YARN-5654
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: Yufei Gu
> Attachments: YARN-5654.002.patch, YARN-5654.003.patch,
> YARN-5654.1.patch
>
>
> With the config:
>
[jira] [Commented] (YARN-6352) Header injections are possible in the application proxy servlet
[
https://issues.apache.org/jira/browse/YARN-6352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15940955#comment-15940955
]
Hadoop QA commented on YARN-6352:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
25s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
55s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
13s{color} | {color:green} branch-2 passed with JDK v1.8.0_121 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
18s{color} | {color:green} branch-2 passed with JDK v1.7.0_121 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
21s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
29s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
12s{color} | {color:green} branch-2 passed with JDK v1.8.0_121 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
14s{color} | {color:green} branch-2 passed with JDK v1.7.0_121 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
11s{color} | {color:green} the patch passed with JDK v1.8.0_121 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
14s{color} | {color:green} the patch passed with JDK v1.7.0_121 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
9s{color} | {color:green} the patch passed with JDK v1.8.0_121 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
12s{color} | {color:green} the patch passed with JDK v1.7.0_121 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
26s{color} | {color:green} hadoop-yarn-server-web-proxy in the patch passed
with JDK v1.7.0_121. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 14m 43s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:b59b8b7 |
| JIRA Issue | YARN-6352 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12860420/YARN-6352-branch-2.003.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux c30284be9fcc 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6
15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | branch-2 / bbd08bb |
| Default Java | 1.7.0_121 |
| Multi-JDK versions | /usr/lib/jvm/java-8-oracle:1.8.0_121
/usr/lib/jvm/java-7-openjdk-amd64:1.7.0_121 |
| findbugs | v3.0.0 |
| JDK v1.7.0_121 Test Results |
[jira] [Comment Edited] (YARN-6384) Add configuratin to set max cpu usage when strict-resource-usage is false with cgroups
[ https://issues.apache.org/jira/browse/YARN-6384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15940948#comment-15940948 ] Miklos Szegedi edited comment on YARN-6384 at 3/24/17 7:00 PM: --- Thank you for reporting this [~lolee_k]. Is not this a duplicate of YARN-5936? was (Author: miklos.szeg...@cloudera.com): Thank you for reporting this [~dengkai]. Is not this a duplicate of YARN-5936? > Add configuratin to set max cpu usage when strict-resource-usage is false > with cgroups > -- > > Key: YARN-6384 > URL: https://issues.apache.org/jira/browse/YARN-6384 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: dengkai > > When using cgroups on yarn, if > yarn.nodemanager.linux-container-executor.cgroups.strict-resource-usage is > false, user may get very more cpu time than expected based on the vcores. > There should be a upper limit even resource-usage is not strict, just like a > percentage which user can get more than promised by vcores. I think it's > important in a shared cluster. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
