[jira] [Commented] (YARN-5783) Verify applications are identified starved
[ https://issues.apache.org/jira/browse/YARN-5783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15649546#comment-15649546 ] Karthik Kambatla commented on YARN-5783: [~templedf] - thanks for the careful reviews and final +1. Let me go ahead and check this in now, so I can trigger the build for YARN-5819 and get it ready for your review. [~leftnoteasy] - filed YARN-5863 to track the addition of starved state. Checking this in.. > Verify applications are identified starved > -- > > Key: YARN-5783 > URL: https://issues.apache.org/jira/browse/YARN-5783 > Project: Hadoop YARN > Issue Type: Sub-task > Components: fairscheduler >Affects Versions: 2.8.0 >Reporter: Karthik Kambatla >Assignee: Karthik Kambatla > Labels: oct16-medium > Attachments: yarn-5783.YARN-4752.1.patch, > yarn-5783.YARN-4752.2.patch, yarn-5783.YARN-4752.3.patch, > yarn-5783.YARN-4752.4.patch, yarn-5783.YARN-4752.5.patch, > yarn-5783.YARN-4752.6.patch, yarn-5783.YARN-4752.7.patch, > yarn-5783.YARN-4752.8.patch > > > JIRA to track unit tests to verify the identification of starved > applications. An application should be marked starved only when: > # Cluster allocation is over the configured threshold for preemption. > # Preemption is enabled for a queue and any of the following: > ## The queue is under its minshare for longer than minsharePreemptionTimeout > ## One of the queue’s applications is under its fairshare for longer than > fairsharePreemptionTimeout. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5783) Verify applications are identified starved
[ https://issues.apache.org/jira/browse/YARN-5783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15649382#comment-15649382 ] Karthik Kambatla commented on YARN-5783: bq. Overall, I think "starved" should be a scheduler-agnostic scheduling state of an app. [~leftnoteasy] - that is an interesting thought. In the current patch for FairScheduler preemption, strictly speaking we are not essentially marking the application as starved. Marking them so makes it easier to handle any changes to the starvation much more gracefully. Given it is related, but not necessary, I would like to follow up on that separately. We should also think carefully about adding scheduler-only states for applications. [~templedf] - mind taking another look? > Verify applications are identified starved > -- > > Key: YARN-5783 > URL: https://issues.apache.org/jira/browse/YARN-5783 > Project: Hadoop YARN > Issue Type: Sub-task > Components: fairscheduler >Affects Versions: 2.8.0 >Reporter: Karthik Kambatla >Assignee: Karthik Kambatla > Labels: oct16-medium > Attachments: yarn-5783.YARN-4752.1.patch, > yarn-5783.YARN-4752.2.patch, yarn-5783.YARN-4752.3.patch, > yarn-5783.YARN-4752.4.patch, yarn-5783.YARN-4752.5.patch, > yarn-5783.YARN-4752.6.patch, yarn-5783.YARN-4752.7.patch, > yarn-5783.YARN-4752.8.patch > > > JIRA to track unit tests to verify the identification of starved > applications. An application should be marked starved only when: > # Cluster allocation is over the configured threshold for preemption. > # Preemption is enabled for a queue and any of the following: > ## The queue is under its minshare for longer than minsharePreemptionTimeout > ## One of the queue’s applications is under its fairshare for longer than > fairsharePreemptionTimeout. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5783) Verify applications are identified starved
[ https://issues.apache.org/jira/browse/YARN-5783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15649334#comment-15649334 ] Daniel Templeton commented on YARN-5783: As with almost everything in FS and CS, the concepts exist in both and ideally could be abstracted into common code, but the devil is in the details. Sounds like good future work. +1 for the latest patch. I'll commit tomorrow morning. > Verify applications are identified starved > -- > > Key: YARN-5783 > URL: https://issues.apache.org/jira/browse/YARN-5783 > Project: Hadoop YARN > Issue Type: Sub-task > Components: fairscheduler >Affects Versions: 2.8.0 >Reporter: Karthik Kambatla >Assignee: Karthik Kambatla > Labels: oct16-medium > Attachments: yarn-5783.YARN-4752.1.patch, > yarn-5783.YARN-4752.2.patch, yarn-5783.YARN-4752.3.patch, > yarn-5783.YARN-4752.4.patch, yarn-5783.YARN-4752.5.patch, > yarn-5783.YARN-4752.6.patch, yarn-5783.YARN-4752.7.patch, > yarn-5783.YARN-4752.8.patch > > > JIRA to track unit tests to verify the identification of starved > applications. An application should be marked starved only when: > # Cluster allocation is over the configured threshold for preemption. > # Preemption is enabled for a queue and any of the following: > ## The queue is under its minshare for longer than minsharePreemptionTimeout > ## One of the queue’s applications is under its fairshare for longer than > fairsharePreemptionTimeout. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5783) Verify applications are identified starved
[
https://issues.apache.org/jira/browse/YARN-5783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15649109#comment-15649109
]
Hadoop QA commented on YARN-5783:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
22s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
27s{color} | {color:green} YARN-4752 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
32s{color} | {color:green} YARN-4752 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
21s{color} | {color:green} YARN-4752 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
38s{color} | {color:green} YARN-4752 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s{color} | {color:green} YARN-4752 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
58s{color} | {color:green} YARN-4752 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
21s{color} | {color:green} YARN-4752 passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 19s{color} | {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 5 new + 85 unchanged - 0 fixed = 90 total (was 85) {color}
|
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 40m
19s{color} | {color:green} hadoop-yarn-server-resourcemanager in the patch
passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 56m 16s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Issue | YARN-5783 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12838054/yarn-5783.YARN-4752.8.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux de6025a72f05 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12
17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | YARN-4752 / b425ca2 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/13831/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/13831/testReport/ |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/13831/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Verify applications are identified starved
> --
>
> Key: YARN-5783
[jira] [Commented] (YARN-5783) Verify applications are identified starved
[ https://issues.apache.org/jira/browse/YARN-5783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15648966#comment-15648966 ] Wangda Tan commented on YARN-5783: -- [~kasha], [~templedf], Haven't looked at the patch, and not sure if you thought this before: could we make the "app starvation level" as an general information which we can get from different scheduler on YARN. Which can help 1) users to understand apps get starved 2) scheduler can make decisions such as preemption. Overall, I think "starved" should be a scheduler-agnostic scheduling state of an app. I'm not trying to stop what you're doing now, please go ahead with what you have, I just wanted to share some ideas. > Verify applications are identified starved > -- > > Key: YARN-5783 > URL: https://issues.apache.org/jira/browse/YARN-5783 > Project: Hadoop YARN > Issue Type: Sub-task > Components: fairscheduler >Affects Versions: 2.8.0 >Reporter: Karthik Kambatla >Assignee: Karthik Kambatla > Labels: oct16-medium > Attachments: yarn-5783.YARN-4752.1.patch, > yarn-5783.YARN-4752.2.patch, yarn-5783.YARN-4752.3.patch, > yarn-5783.YARN-4752.4.patch, yarn-5783.YARN-4752.5.patch, > yarn-5783.YARN-4752.6.patch, yarn-5783.YARN-4752.7.patch, > yarn-5783.YARN-4752.8.patch > > > JIRA to track unit tests to verify the identification of starved > applications. An application should be marked starved only when: > # Cluster allocation is over the configured threshold for preemption. > # Preemption is enabled for a queue and any of the following: > ## The queue is under its minshare for longer than minsharePreemptionTimeout > ## One of the queue’s applications is under its fairshare for longer than > fairsharePreemptionTimeout. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5783) Verify applications are identified starved
[
https://issues.apache.org/jira/browse/YARN-5783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15646513#comment-15646513
]
Hadoop QA commented on YARN-5783:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
29s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
4s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
24s{color} | {color:green} YARN-4752 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
26s{color} | {color:green} YARN-4752 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s{color} | {color:green} YARN-4752 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
11s{color} | {color:green} YARN-4752 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
32s{color} | {color:green} YARN-4752 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
6s{color} | {color:green} YARN-4752 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
52s{color} | {color:green} YARN-4752 passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
11s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
24s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 43s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch
generated 5 new + 88 unchanged - 0 fixed = 93 total (was 88) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
11s{color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
generated 3 new + 0 unchanged - 0 fixed = 3 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 2m 46s{color}
| {color:red} hadoop-yarn-common in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 33m 22s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
35s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 70m 33s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common |
| |
org.apache.hadoop.yarn.api.records.impl.pb.ApplicationAttemptIdPBImpl.equals(Object)
checks for operand being a
org.apache.hadoop.yarn.proto.YarnProtos$ApplicationAttemptIdProto$Builder At
ApplicationAttemptIdPBImpl.java:a
org.apache.hadoop.yarn.proto.YarnProtos$ApplicationAttemptIdProto$Builder At
ApplicationAttemptIdPBImpl.java:[line 100] |
| |
org.apache.hadoop.yarn.api.records.impl.pb.ApplicationAttemptIdPBImpl.equals(Object)
checks for operand being a
org.apache.hadoop.yarn.proto.YarnProtos$ApplicationAttemptIdProto At
ApplicationAttemptIdPBImpl.java:a
org.apache.hadoop.yarn.proto.YarnProtos$ApplicationAttemptIdProto At
ApplicationAttemptIdPBImpl.java:[line 96] |
| |
org.apache.hadoop.yarn.api.records.impl.pb.ApplicationAttemptIdPBImpl.equals(Object)
is unusual At
ApplicationAttemptIdPBImpl.java:ApplicationAttemptIdPBImpl.java:[lines 92-104] |
| Failed junit
[jira] [Commented] (YARN-5783) Verify applications are identified starved
[
https://issues.apache.org/jira/browse/YARN-5783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15646386#comment-15646386
]
Karthik Kambatla commented on YARN-5783:
Fair point. That was actually my initial thought before googling around.
Updated patch uses {{getApplicationAttemptId().hashCode()}}. However, I
realized ApplicationAttemptIdPBImpl doesn't implement hashCode even if the
proto itself does. So, implemented that using the proto.
> Verify applications are identified starved
> --
>
> Key: YARN-5783
> URL: https://issues.apache.org/jira/browse/YARN-5783
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: fairscheduler
>Affects Versions: 2.8.0
>Reporter: Karthik Kambatla
>Assignee: Karthik Kambatla
> Labels: oct16-medium
> Attachments: yarn-5783.YARN-4752.1.patch,
> yarn-5783.YARN-4752.2.patch, yarn-5783.YARN-4752.3.patch,
> yarn-5783.YARN-4752.4.patch, yarn-5783.YARN-4752.5.patch,
> yarn-5783.YARN-4752.6.patch, yarn-5783.YARN-4752.7.patch
>
>
> JIRA to track unit tests to verify the identification of starved
> applications. An application should be marked starved only when:
> # Cluster allocation is over the configured threshold for preemption.
> # Preemption is enabled for a queue and any of the following:
> ## The queue is under its minshare for longer than minsharePreemptionTimeout
> ## One of the queue’s applications is under its fairshare for longer than
> fairsharePreemptionTimeout.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5783) Verify applications are identified starved
[
https://issues.apache.org/jira/browse/YARN-5783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15645749#comment-15645749
]
Daniel Templeton commented on YARN-5783:
According to your Stackoverflow link, the correct implementation is:
{code}
@Override
public int hashCode() {
return Objects.hash(getApplicationAttemptId());
}
{code}
I take issue with even that approach, though. If the only thing that makes the
hash code different between two attempts is the hash code of the attempt ID
(which is sane and makes logical sense), then the {{hashCode()}} method should
return {{getApplicationAttemptId().hashCode()}}.
Other than that, I'm happy with the latest patch.
> Verify applications are identified starved
> --
>
> Key: YARN-5783
> URL: https://issues.apache.org/jira/browse/YARN-5783
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: fairscheduler
>Affects Versions: 2.8.0
>Reporter: Karthik Kambatla
>Assignee: Karthik Kambatla
> Labels: oct16-medium
> Attachments: yarn-5783.YARN-4752.1.patch,
> yarn-5783.YARN-4752.2.patch, yarn-5783.YARN-4752.3.patch,
> yarn-5783.YARN-4752.4.patch, yarn-5783.YARN-4752.5.patch,
> yarn-5783.YARN-4752.6.patch
>
>
> JIRA to track unit tests to verify the identification of starved
> applications. An application should be marked starved only when:
> # Cluster allocation is over the configured threshold for preemption.
> # Preemption is enabled for a queue and any of the following:
> ## The queue is under its minshare for longer than minsharePreemptionTimeout
> ## One of the queue’s applications is under its fairshare for longer than
> fairsharePreemptionTimeout.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5783) Verify applications are identified starved
[
https://issues.apache.org/jira/browse/YARN-5783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15635536#comment-15635536
]
Hadoop QA commented on YARN-5783:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
21s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 11m
55s{color} | {color:green} YARN-4752 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
37s{color} | {color:green} YARN-4752 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
28s{color} | {color:green} YARN-4752 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
43s{color} | {color:green} YARN-4752 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
21s{color} | {color:green} YARN-4752 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
8s{color} | {color:green} YARN-4752 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
24s{color} | {color:green} YARN-4752 passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
32s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 21s{color} | {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 5 new + 86 unchanged - 0 fixed = 91 total (was 86) {color}
|
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 38m
10s{color} | {color:green} hadoop-yarn-server-resourcemanager in the patch
passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 59m 45s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Issue | YARN-5783 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12837058/yarn-5783.YARN-4752.6.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux f3ef4d56c029 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18
21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | YARN-4752 / b425ca2 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/13781/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/13781/testReport/ |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/13781/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Verify applications are identified starved
> --
>
> Key: YARN-5783
[jira] [Commented] (YARN-5783) Verify applications are identified starved
[ https://issues.apache.org/jira/browse/YARN-5783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15634830#comment-15634830 ] Karthik Kambatla commented on YARN-5783: bq. In SchedulerApplicationAttempt.hashCode(), what is the value of adding the prime * positive? Honestly, not sure. This [stackoverflow link|http://stackoverflow.com/questions/113511/best-implementation-for-hashcode-method] suggested it, and all auto-generated protobuf files seem to adopt that approach too. Addressing the other two comments in the next patch. > Verify applications are identified starved > -- > > Key: YARN-5783 > URL: https://issues.apache.org/jira/browse/YARN-5783 > Project: Hadoop YARN > Issue Type: Sub-task > Components: fairscheduler >Affects Versions: 2.8.0 >Reporter: Karthik Kambatla >Assignee: Karthik Kambatla > Labels: oct16-medium > Attachments: yarn-5783.YARN-4752.1.patch, > yarn-5783.YARN-4752.2.patch, yarn-5783.YARN-4752.3.patch, > yarn-5783.YARN-4752.4.patch, yarn-5783.YARN-4752.5.patch > > > JIRA to track unit tests to verify the identification of starved > applications. An application should be marked starved only when: > # Cluster allocation is over the configured threshold for preemption. > # Preemption is enabled for a queue and any of the following: > ## The queue is under its minshare for longer than minsharePreemptionTimeout > ## One of the queue’s applications is under its fairshare for longer than > fairsharePreemptionTimeout. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5783) Verify applications are identified starved
[
https://issues.apache.org/jira/browse/YARN-5783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15634725#comment-15634725
]
Hadoop QA commented on YARN-5783:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
24s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
51s{color} | {color:green} YARN-4752 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
41s{color} | {color:green} YARN-4752 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
26s{color} | {color:green} YARN-4752 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
46s{color} | {color:green} YARN-4752 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
17s{color} | {color:green} YARN-4752 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
2s{color} | {color:green} YARN-4752 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
25s{color} | {color:green} YARN-4752 passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 22s{color} | {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 5 new + 85 unchanged - 0 fixed = 90 total (was 85) {color}
|
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 37m 57s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 56m 41s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.yarn.server.resourcemanager.TestRMHA |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Issue | YARN-5783 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12836728/yarn-5783.YARN-4752.5.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux c700ec1bcc86 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | YARN-4752 / b425ca2 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/13774/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
|
| unit |
https://builds.apache.org/job/PreCommit-YARN-Build/13774/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/13774/testReport/ |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
|
| Console output |
[jira] [Commented] (YARN-5783) Verify applications are identified starved
[
https://issues.apache.org/jira/browse/YARN-5783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15634515#comment-15634515
]
Daniel Templeton commented on YARN-5783:
Comments:
* In {{SchedulerApplicationAttempt.hashCode()}}, what is the value of adding
the prime \* positive?
* In {{FSStarvedApps.take()}} javadoc you don't need the hyphen in the @throws
message.
* In {{TestFSAppsStarvation}} test methods, should {code}
setupStarvedCluster();
submitAppsToEachLeafQueue();
sendNodeUpdateEvents();
// Sleep to hit the preemption timeouts
Thread.sleep(10);
// Scheduler update to populate starved apps
scheduler.update();
// Wait for apps to be processed by MockPreemptionThread
Thread.yield();{code} be put into a common method to be called from all the
test methods? And maybe call that from {{testPreemptionDisabled()}} as well so
that you give it a fair chance to prove that it really is disabled?
> Verify applications are identified starved
> --
>
> Key: YARN-5783
> URL: https://issues.apache.org/jira/browse/YARN-5783
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: fairscheduler
>Affects Versions: 2.8.0
>Reporter: Karthik Kambatla
>Assignee: Karthik Kambatla
> Labels: oct16-medium
> Attachments: yarn-5783.YARN-4752.1.patch,
> yarn-5783.YARN-4752.2.patch, yarn-5783.YARN-4752.3.patch,
> yarn-5783.YARN-4752.4.patch, yarn-5783.YARN-4752.5.patch
>
>
> JIRA to track unit tests to verify the identification of starved
> applications. An application should be marked starved only when:
> # Cluster allocation is over the configured threshold for preemption.
> # Preemption is enabled for a queue and any of the following:
> ## The queue is under its minshare for longer than minsharePreemptionTimeout
> ## One of the queue’s applications is under its fairshare for longer than
> fairsharePreemptionTimeout.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5783) Verify applications are identified starved
[ https://issues.apache.org/jira/browse/YARN-5783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15631629#comment-15631629 ] Karthik Kambatla commented on YARN-5783: Good points, [~templedf]. Your last suggestion got me thinking. It seemed like mocking the preemption thread to consume starved apps but not actually preempt would allow us to test starvation logic better. Did the same. We could potentially add more advanced tests to verify starvation based on thresholds and timeouts in a subsequent JIRA (YARN-5824). By the way, this patch conflicts with YARN-5821 which is mostly cosmetic changes and I believe is ready to go. > Verify applications are identified starved > -- > > Key: YARN-5783 > URL: https://issues.apache.org/jira/browse/YARN-5783 > Project: Hadoop YARN > Issue Type: Sub-task > Components: fairscheduler >Affects Versions: 2.8.0 >Reporter: Karthik Kambatla >Assignee: Karthik Kambatla > Labels: oct16-medium > Attachments: yarn-5783.YARN-4752.1.patch, > yarn-5783.YARN-4752.2.patch, yarn-5783.YARN-4752.3.patch, > yarn-5783.YARN-4752.4.patch, yarn-5783.YARN-4752.5.patch > > > JIRA to track unit tests to verify the identification of starved > applications. An application should be marked starved only when: > # Cluster allocation is over the configured threshold for preemption. > # Preemption is enabled for a queue and any of the following: > ## The queue is under its minshare for longer than minsharePreemptionTimeout > ## One of the queue’s applications is under its fairshare for longer than > fairsharePreemptionTimeout. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
