[jira] [Updated] (YARN-6959) RM may allocate wrong AM Container for new attempt
[
https://issues.apache.org/jira/browse/YARN-6959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yuqi Wang updated YARN-6959:
Attachment: (was: YARN-6959.004.patch)
> RM may allocate wrong AM Container for new attempt
> --
>
> Key: YARN-6959
> URL: https://issues.apache.org/jira/browse/YARN-6959
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler, fairscheduler, scheduler
>Affects Versions: 2.7.1
>Reporter: Yuqi Wang
>Assignee: Yuqi Wang
> Labels: patch
> Fix For: 2.7.1, 3.0.0-alpha4
>
> Attachments: YARN-6959.001.patch, YARN-6959.002.patch,
> YARN-6959.003.patch, YARN-6959.004.patch, YARN-6959-branch-2.7.001.patch
>
>
> *Issue Summary:*
> Previous attempt ResourceRequest may be recorded into current attempt
> ResourceRequests. These mis-recorded ResourceRequests may confuse AM
> Container Request and Allocation for current attempt.
> *Issue Pipeline:*
> {code:java}
> // Executing precondition check for the incoming attempt id.
> ApplicationMasterService.allocate() ->
> scheduler.allocate(attemptId, ask, ...) ->
> // Previous precondition check for the attempt id may be outdated here,
> // i.e. the currentAttempt may not be the corresponding attempt of the
> attemptId.
> // Such as the attempt id is corresponding to the previous attempt.
> currentAttempt = scheduler.getApplicationAttempt(attemptId) ->
> // Previous attempt ResourceRequest may be recorded into current attempt
> ResourceRequests
> currentAttempt.updateResourceRequests(ask) ->
> // RM may allocate wrong AM Container for the current attempt, because its
> ResourceRequests
> // may come from previous attempt which can be any ResourceRequests previous
> AM asked
> // and there is not matching logic for the original AM Container
> ResourceRequest and
> // the returned amContainerAllocation below.
> AMContainerAllocatedTransition.transition(...) ->
> amContainerAllocation = scheduler.allocate(currentAttemptId, ...)
> {code}
> *Patch Correctness:*
> Because after this Patch, RM will definitely record ResourceRequests from
> different attempt into different objects of
> SchedulerApplicationAttempt.AppSchedulingInfo.
> So, even if RM still record ResourceRequests from old attempt at any time,
> these ResourceRequests will be recorded in old AppSchedulingInfo object which
> will not impact current attempt's resource requests and allocation.
> *Concerns:*
> The getApplicationAttempt function in AbstractYarnScheduler is so confusing,
> we should better rename it to getCurrentApplicationAttempt. And reconsider
> whether there are any other bugs related to getApplicationAttempt.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6959) RM may allocate wrong AM Container for new attempt
[
https://issues.apache.org/jira/browse/YARN-6959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yuqi Wang updated YARN-6959:
Attachment: YARN-6959.004.patch
> RM may allocate wrong AM Container for new attempt
> --
>
> Key: YARN-6959
> URL: https://issues.apache.org/jira/browse/YARN-6959
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler, fairscheduler, scheduler
>Affects Versions: 2.7.1
>Reporter: Yuqi Wang
>Assignee: Yuqi Wang
> Labels: patch
> Fix For: 2.7.1, 3.0.0-alpha4
>
> Attachments: YARN-6959.001.patch, YARN-6959.002.patch,
> YARN-6959.003.patch, YARN-6959.004.patch, YARN-6959-branch-2.7.001.patch
>
>
> *Issue Summary:*
> Previous attempt ResourceRequest may be recorded into current attempt
> ResourceRequests. These mis-recorded ResourceRequests may confuse AM
> Container Request and Allocation for current attempt.
> *Issue Pipeline:*
> {code:java}
> // Executing precondition check for the incoming attempt id.
> ApplicationMasterService.allocate() ->
> scheduler.allocate(attemptId, ask, ...) ->
> // Previous precondition check for the attempt id may be outdated here,
> // i.e. the currentAttempt may not be the corresponding attempt of the
> attemptId.
> // Such as the attempt id is corresponding to the previous attempt.
> currentAttempt = scheduler.getApplicationAttempt(attemptId) ->
> // Previous attempt ResourceRequest may be recorded into current attempt
> ResourceRequests
> currentAttempt.updateResourceRequests(ask) ->
> // RM may allocate wrong AM Container for the current attempt, because its
> ResourceRequests
> // may come from previous attempt which can be any ResourceRequests previous
> AM asked
> // and there is not matching logic for the original AM Container
> ResourceRequest and
> // the returned amContainerAllocation below.
> AMContainerAllocatedTransition.transition(...) ->
> amContainerAllocation = scheduler.allocate(currentAttemptId, ...)
> {code}
> *Patch Correctness:*
> Because after this Patch, RM will definitely record ResourceRequests from
> different attempt into different objects of
> SchedulerApplicationAttempt.AppSchedulingInfo.
> So, even if RM still record ResourceRequests from old attempt at any time,
> these ResourceRequests will be recorded in old AppSchedulingInfo object which
> will not impact current attempt's resource requests and allocation.
> *Concerns:*
> The getApplicationAttempt function in AbstractYarnScheduler is so confusing,
> we should better rename it to getCurrentApplicationAttempt. And reconsider
> whether there are any other bugs related to getApplicationAttempt.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6903) Yarn-native-service framework core rewrite
[
https://issues.apache.org/jira/browse/YARN-6903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16117851#comment-16117851
]
Hadoop QA commented on YARN-6903:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
24s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 60 new or modified test
files. {color} |
|| || || || {color:brown} yarn-native-services Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 3m
31s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
24s{color} | {color:green} yarn-native-services passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m
32s{color} | {color:green} yarn-native-services passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
49s{color} | {color:green} yarn-native-services passed {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
33s{color} | {color:red} hadoop-yarn-common in yarn-native-services failed.
{color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
35s{color} | {color:red} hadoop-yarn-server-resourcemanager in
yarn-native-services failed. {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
30s{color} | {color:red} hadoop-yarn-slider in yarn-native-services failed.
{color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
27s{color} | {color:red} hadoop-yarn-slider-core in yarn-native-services
failed. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-slider
{color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
22s{color} | {color:red} hadoop-yarn-common in yarn-native-services failed.
{color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
26s{color} | {color:red} hadoop-yarn-server-resourcemanager in
yarn-native-services failed. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
26s{color} | {color:red} hadoop-yarn-slider-core in yarn-native-services
failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 4m
34s{color} | {color:green} yarn-native-services passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
13s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 6m
42s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 6m 42s{color}
| {color:red} hadoop-yarn-project_hadoop-yarn generated 3 new + 133 unchanged -
5 fixed = 136 total (was 138) {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 31s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch
generated 518 new + 1522 unchanged - 403 fixed = 2040 total (was 1925) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 7m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} shellcheck {color} | {color:green} 0m
24s{color} | {color:green} There were no new shellcheck issues. {color} |
| {color:green}+1{color} | {color:green} shelldocs {color} | {color:green} 0m
11s{color} | {color:green} There were no new shelldocs issues. {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 26 line(s) that end in whitespace. Use
git apply --whitespace=fix <>. Refer
https://git-scm.com/docs/git-apply {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
1s{color} | {color:red} The patch 1 line(s) with tabs. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
5s{color} |
[jira] [Commented] (YARN-6959) RM may allocate wrong AM Container for new attempt
[
https://issues.apache.org/jira/browse/YARN-6959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16117829#comment-16117829
]
Hadoop QA commented on YARN-6959:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
15s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
7s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
34s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
5s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
20s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
26s{color} | {color:green}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 0 new + 390 unchanged - 4 fixed = 390 total (was 394)
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 45m 52s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 68m 23s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.resourcemanager.scheduler.fair.TestFSAppStarvation |
| | hadoop.yarn.server.resourcemanager.TestRMEmbeddedElector |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | YARN-6959 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12880777/YARN-6959.004.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 916222ef4bd3 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12
13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 8d3fd81 |
| Default Java | 1.8.0_131 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-YARN-Build/16766/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/16766/testReport/ |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/16766/console |
| Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> RM may allocate wrong AM Container for new attempt
>
[jira] [Commented] (YARN-6965) Duplicate instantiation in FairSchedulerQueueInfo
[ https://issues.apache.org/jira/browse/YARN-6965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16117824#comment-16117824 ] Masahiro Tanaka commented on YARN-6965: --- I would like to fix this duplication. Could anyone assign me to this issue? > Duplicate instantiation in FairSchedulerQueueInfo > - > > Key: YARN-6965 > URL: https://issues.apache.org/jira/browse/YARN-6965 > Project: Hadoop YARN > Issue Type: Bug > Components: fairscheduler >Affects Versions: 3.0.0-alpha3 >Reporter: Masahiro Tanaka >Priority: Minor > > There is a duplicate instantiation in FairSchedulerQueueInfo.java > https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/FairSchedulerQueueInfo.java#L102-L105 > I think this is not a big issue, but we should fix this in order to avoid > confusion. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-6965) Duplicate instantiation in FairSchedulerQueueInfo
Masahiro Tanaka created YARN-6965: - Summary: Duplicate instantiation in FairSchedulerQueueInfo Key: YARN-6965 URL: https://issues.apache.org/jira/browse/YARN-6965 Project: Hadoop YARN Issue Type: Bug Components: fairscheduler Affects Versions: 3.0.0-alpha3 Reporter: Masahiro Tanaka Priority: Minor There is a duplicate instantiation in FairSchedulerQueueInfo.java https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/FairSchedulerQueueInfo.java#L102-L105 I think this is not a big issue, but we should fix this in order to avoid confusion. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6940) FairScheduler: Enable Container update CodePaths and container resize testcase
[
https://issues.apache.org/jira/browse/YARN-6940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16117821#comment-16117821
]
Hadoop QA commented on YARN-6940:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
15s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
11s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
57s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
45s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 5m
14s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 55s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch
generated 1 new + 81 unchanged - 0 fixed = 82 total (was 81) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 46m 34s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 20m
16s{color} | {color:green} hadoop-yarn-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
29s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}113m 57s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.resourcemanager.scheduler.fair.TestFSAppStarvation |
| | hadoop.yarn.server.resourcemanager.TestRMEmbeddedElector |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | YARN-6940 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12880236/YARN-6940.001.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux aecec9259960 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12
13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 8d3fd81 |
| Default Java | 1.8.0_131 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/16764/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
|
| unit |
https://builds.apache.org/job/PreCommit-YARN-Build/16764/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
|
| Test
[jira] [Updated] (YARN-6959) RM may allocate wrong AM Container for new attempt
[
https://issues.apache.org/jira/browse/YARN-6959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yuqi Wang updated YARN-6959:
Component/s: fairscheduler
capacity scheduler
> RM may allocate wrong AM Container for new attempt
> --
>
> Key: YARN-6959
> URL: https://issues.apache.org/jira/browse/YARN-6959
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler, fairscheduler, scheduler
>Affects Versions: 2.7.1
>Reporter: Yuqi Wang
>Assignee: Yuqi Wang
> Labels: patch
> Fix For: 2.7.1, 3.0.0-alpha4
>
> Attachments: YARN-6959.001.patch, YARN-6959.002.patch,
> YARN-6959.003.patch, YARN-6959.004.patch, YARN-6959-branch-2.7.001.patch
>
>
> *Issue Summary:*
> Previous attempt ResourceRequest may be recorded into current attempt
> ResourceRequests. These mis-recorded ResourceRequests may confuse AM
> Container Request and Allocation for current attempt.
> *Issue Pipeline:*
> {code:java}
> // Executing precondition check for the incoming attempt id.
> ApplicationMasterService.allocate() ->
> scheduler.allocate(attemptId, ask, ...) ->
> // Previous precondition check for the attempt id may be outdated here,
> // i.e. the currentAttempt may not be the corresponding attempt of the
> attemptId.
> // Such as the attempt id is corresponding to the previous attempt.
> currentAttempt = scheduler.getApplicationAttempt(attemptId) ->
> // Previous attempt ResourceRequest may be recorded into current attempt
> ResourceRequests
> currentAttempt.updateResourceRequests(ask) ->
> // RM may allocate wrong AM Container for the current attempt, because its
> ResourceRequests
> // may come from previous attempt which can be any ResourceRequests previous
> AM asked
> // and there is not matching logic for the original AM Container
> ResourceRequest and
> // the returned amContainerAllocation below.
> AMContainerAllocatedTransition.transition(...) ->
> amContainerAllocation = scheduler.allocate(currentAttemptId, ...)
> {code}
> *Patch Correctness:*
> Because after this Patch, RM will definitely record ResourceRequests from
> different attempt into different objects of
> SchedulerApplicationAttempt.AppSchedulingInfo.
> So, even if RM still record ResourceRequests from old attempt at any time,
> these ResourceRequests will be recorded in old AppSchedulingInfo object which
> will not impact current attempt's resource requests and allocation.
> *Concerns:*
> The getApplicationAttempt function in AbstractYarnScheduler is so confusing,
> we should better rename it to getCurrentApplicationAttempt. And reconsider
> whether there are any other bugs related to getApplicationAttempt.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6959) RM may allocate wrong AM Container for new attempt
[
https://issues.apache.org/jira/browse/YARN-6959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16117796#comment-16117796
]
Hadoop QA commented on YARN-6959:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
34s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
24s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 28s{color} | {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 1 new + 393 unchanged - 1 fixed = 394 total (was 394)
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 44m
28s{color} | {color:green} hadoop-yarn-server-resourcemanager in the patch
passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 69m 19s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | YARN-6959 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12880771/YARN-6959.003.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 493a53b5cebf 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31
14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 8d3fd81 |
| Default Java | 1.8.0_131 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/16765/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/16765/testReport/ |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/16765/console |
| Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> RM may allocate wrong AM Container for new attempt
> --
>
> Key: YARN-6959
> URL: https://issues.apache.org/jira/browse/YARN-6959
>
[jira] [Commented] (YARN-6811) [ATS1.5] All history logs should be kept under its own User Directory.
[ https://issues.apache.org/jira/browse/YARN-6811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16117790#comment-16117790 ] Rohith Sharma K S commented on YARN-6811: - cc :/ [~djp] Updated the patch for branch-2 > [ATS1.5] All history logs should be kept under its own User Directory. > --- > > Key: YARN-6811 > URL: https://issues.apache.org/jira/browse/YARN-6811 > Project: Hadoop YARN > Issue Type: Improvement > Components: timelineclient, timelineserver >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S > Attachments: YARN-6811.01.patch, YARN-6811.02.patch, > YARN-6811-branch-2.01.patch > > > ATS1.5 allows to store history data in underlying FileSystem folder path i.e > */acitve-dir* and */done-dir*. These base directories are protected for > unauthorized user access for other users data by setting sticky bit for > /active-dir. > But object store filesystems such as WASB does not have user access control > on folders and files. When WASB are used as underlying file system for > ATS1.5, the history data which are stored in FS are accessible to all users. > *This would be a security risk* > I would propose to keep history data under its own user directory i.e > */active-dir/$USER*. Even this do not solve basic user access from FS, but it > provides capability to plugin Apache Ranger policies for each user folders. > One thing to note that setting policies to each user folder is admin > responsibility. But grouping all history data of one user folder allows to > set policies so that user access control is achieved. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6959) RM may allocate wrong AM Container for new attempt
[
https://issues.apache.org/jira/browse/YARN-6959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yuqi Wang updated YARN-6959:
Attachment: YARN-6959.004.patch
Adjust Style
> RM may allocate wrong AM Container for new attempt
> --
>
> Key: YARN-6959
> URL: https://issues.apache.org/jira/browse/YARN-6959
> Project: Hadoop YARN
> Issue Type: Bug
> Components: scheduler
>Affects Versions: 2.7.1
>Reporter: Yuqi Wang
>Assignee: Yuqi Wang
> Labels: patch
> Fix For: 2.7.1, 3.0.0-alpha4
>
> Attachments: YARN-6959.001.patch, YARN-6959.002.patch,
> YARN-6959.003.patch, YARN-6959.004.patch, YARN-6959-branch-2.7.001.patch
>
>
> *Issue Summary:*
> Previous attempt ResourceRequest may be recorded into current attempt
> ResourceRequests. These mis-recorded ResourceRequests may confuse AM
> Container Request and Allocation for current attempt.
> *Issue Pipeline:*
> {code:java}
> // Executing precondition check for the incoming attempt id.
> ApplicationMasterService.allocate() ->
> scheduler.allocate(attemptId, ask, ...) ->
> // Previous precondition check for the attempt id may be outdated here,
> // i.e. the currentAttempt may not be the corresponding attempt of the
> attemptId.
> // Such as the attempt id is corresponding to the previous attempt.
> currentAttempt = scheduler.getApplicationAttempt(attemptId) ->
> // Previous attempt ResourceRequest may be recorded into current attempt
> ResourceRequests
> currentAttempt.updateResourceRequests(ask) ->
> // RM may allocate wrong AM Container for the current attempt, because its
> ResourceRequests
> // may come from previous attempt which can be any ResourceRequests previous
> AM asked
> // and there is not matching logic for the original AM Container
> ResourceRequest and
> // the returned amContainerAllocation below.
> AMContainerAllocatedTransition.transition(...) ->
> amContainerAllocation = scheduler.allocate(currentAttemptId, ...)
> {code}
> *Patch Correctness:*
> Because after this Patch, RM will definitely record ResourceRequests from
> different attempt into different objects of
> SchedulerApplicationAttempt.AppSchedulingInfo.
> So, even if RM still record ResourceRequests from old attempt at any time,
> these ResourceRequests will be recorded in old AppSchedulingInfo object which
> will not impact current attempt's resource requests and allocation.
> *Concerns:*
> The getApplicationAttempt function in AbstractYarnScheduler is so confusing,
> we should better rename it to getCurrentApplicationAttempt. And reconsider
> whether there are any other bugs related to getApplicationAttempt.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6956) preemption may only consider resource requests for one node
[
https://issues.apache.org/jira/browse/YARN-6956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16117772#comment-16117772
]
Karthik Kambatla commented on YARN-6956:
There are a series of follow-up changes to be made:
# Prioritize the RR to check: at a particular priority level, maybe we should
look at the RR with least resources.
# Consider more than one RR. While considering all RRs might be too expensive,
we could consider a configurable number of RRs.
# Honor delay scheduling for preemption as well. If we are unable to identify
containers to preempt on a node and there is a corresponding rack-local RR,
consider finding any node on the rack. Similarly, relax to the entire cluster.
[~Steven Rand], do these suggestions make sense? Will you be interested in
contributing?
> preemption may only consider resource requests for one node
> ---
>
> Key: YARN-6956
> URL: https://issues.apache.org/jira/browse/YARN-6956
> Project: Hadoop YARN
> Issue Type: Bug
> Components: fairscheduler
>Affects Versions: 2.9.0, 3.0.0-beta1
> Environment: CDH 5.11.0
>Reporter: Steven Rand
>
> I'm observing the following series of events on a CDH 5.11.0 cluster, which
> seem to be possible after YARN-6163:
> 1. An application is considered to be starved, so {{FSPreemptionThread}}
> calls {{identifyContainersToPreempt}}, and that calls
> {{FSAppAttempt#getStarvedResourceRequests}} to get a list of
> {{ResourceRequest}} instances that are enough to address the app's starvation.
> 2. The first {{ResourceRequest}} that {{getStarvedResourceRequests}} sees is
> enough to address the app's starvation, so we break out of the loop over
> {{appSchedulingInfo.getAllResourceRequests()}} after only one iteration:
> https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSAppAttempt.java#L1180.
> We return only this one {{ResourceRequest}} back to the
> {{identifyContainersToPreempt}} method.
> 3. It turns out that this particular {{ResourceRequest}} happens to have a
> value for {{getResourceName}} that identifies a specific node in the cluster.
> This causes preemption to only consider containers on that node, and not the
> rest of the cluster.
> [~kasha], does that make sense? I'm happy to submit a patch if I'm
> understanding the problem correctly.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6852) [YARN-6223] Native code changes to support isolate GPU devices by using CGroups
[ https://issues.apache.org/jira/browse/YARN-6852?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16117773#comment-16117773 ] Zhankun Tang commented on YARN-6852: [~miklos.szeg...@cloudera.com], [~wangda], Thanks for the good progress. The patch LGTM. Perhaps one minor reason/benefit we prefer "devices.deny" comparing with "devices.allow" is because a newly added parent group blacklist will propagate to the children while whitelist entries will not? > [YARN-6223] Native code changes to support isolate GPU devices by using > CGroups > --- > > Key: YARN-6852 > URL: https://issues.apache.org/jira/browse/YARN-6852 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan >Assignee: Wangda Tan > Attachments: YARN-6852.001.patch, YARN-6852.002.patch, > YARN-6852.003.patch, YARN-6852.004.patch > > > This JIRA plan to add support of: > 1) Isolation in CGroups. (native side). -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6240) TestCapacityScheduler.testRefreshQueuesWithQueueDelete fails randomly
[
https://issues.apache.org/jira/browse/YARN-6240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16117769#comment-16117769
]
Naganarasimha G R commented on YARN-6240:
-
Thanks [~wangda],
seems like YARN-6741 will go in today, based on that we will be able to
further analyze.
> TestCapacityScheduler.testRefreshQueuesWithQueueDelete fails randomly
> -
>
> Key: YARN-6240
> URL: https://issues.apache.org/jira/browse/YARN-6240
> Project: Hadoop YARN
> Issue Type: Test
> Components: test
>Reporter: Sunil G
>Assignee: Naganarasimha G R
> Attachments: YARN-6240.001.patch
>
>
> *Error Message*
> Expected to NOT throw exception when refresh queue tries to delete a queue
> WITHOUT running apps
> Link
> [here|https://builds.apache.org/job/PreCommit-YARN-Build/15092/testReport/org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity/TestCapacityScheduler/testRefreshQueuesWithQueueDelete/]
> *Stacktrace*
> {code}
> java.lang.AssertionError: Expected to NOT throw exception when refresh queue
> tries to delete a queue WITHOUT running apps
> at org.junit.Assert.fail(Assert.java:88)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler.testRefreshQueuesWithQueueDelete(TestCapacityScheduler.java:3875)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6890) If UI is not secured, we allow user to kill other users' job even yarn cluster is secured.
[
https://issues.apache.org/jira/browse/YARN-6890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16117763#comment-16117763
]
Hadoop QA commented on YARN-6890:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
14s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
2s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 16m
2s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
6s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
10s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
18s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
20s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 11m
57s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
2m 11s{color} | {color:orange} root: The patch generated 4 new + 127 unchanged
- 0 fixed = 131 total (was 127) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
25s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 7m 20s{color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
23s{color} | {color:green} hadoop-yarn-server-common in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
34s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 92m 3s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.net.TestDNS |
| | hadoop.fs.sftp.TestSFTPFileSystem |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | YARN-6890 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12880759/YARN-6890-v3.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 7c0db5a3d5b8 3.13.0-119-generic #166-Ubuntu SMP Wed May 3
12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / c61f2c4 |
| Default Java | 1.8.0_131 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/16763/artifact/patchprocess/diff-checkstyle-root.txt
|
| unit |
https://builds.apache.org/job/PreCommit-YARN-Build/16763/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
|
| Test Results |
[jira] [Commented] (YARN-6964) Fair scheduler misuses Resources operations
[
https://issues.apache.org/jira/browse/YARN-6964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16117761#comment-16117761
]
Hadoop QA commented on YARN-6964:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
48s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m
9s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
5s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
41s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
17s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
10s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
31s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
53s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 48m 38s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
38s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}120m 17s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestIncreaseAllocationExpirer
|
| | hadoop.yarn.server.resourcemanager.monitor.TestSchedulingMonitor |
| |
hadoop.yarn.server.resourcemanager.scheduler.fair.TestAllocationFileLoaderService
|
| | hadoop.yarn.server.resourcemanager.scheduler.fair.TestFSAppStarvation |
| Timed out junit tests |
org.apache.hadoop.yarn.server.resourcemanager.TestSubmitApplicationWithRMHA |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | YARN-6964 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12880754/YARN-6964.001.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 26c83b36be24 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31
14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision |
[jira] [Updated] (YARN-6959) RM may allocate wrong AM Container for new attempt
[
https://issues.apache.org/jira/browse/YARN-6959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yuqi Wang updated YARN-6959:
Attachment: YARN-6959.003.patch
Re-trigger QA use the same patch as 002
> RM may allocate wrong AM Container for new attempt
> --
>
> Key: YARN-6959
> URL: https://issues.apache.org/jira/browse/YARN-6959
> Project: Hadoop YARN
> Issue Type: Bug
> Components: scheduler
>Affects Versions: 2.7.1
>Reporter: Yuqi Wang
>Assignee: Yuqi Wang
> Labels: patch
> Fix For: 2.7.1, 3.0.0-alpha4
>
> Attachments: YARN-6959.001.patch, YARN-6959.002.patch,
> YARN-6959.003.patch, YARN-6959-branch-2.7.001.patch
>
>
> *Issue Summary:*
> Previous attempt ResourceRequest may be recorded into current attempt
> ResourceRequests. These mis-recorded ResourceRequests may confuse AM
> Container Request and Allocation for current attempt.
> *Issue Pipeline:*
> {code:java}
> // Executing precondition check for the incoming attempt id.
> ApplicationMasterService.allocate() ->
> scheduler.allocate(attemptId, ask, ...) ->
> // Previous precondition check for the attempt id may be outdated here,
> // i.e. the currentAttempt may not be the corresponding attempt of the
> attemptId.
> // Such as the attempt id is corresponding to the previous attempt.
> currentAttempt = scheduler.getApplicationAttempt(attemptId) ->
> // Previous attempt ResourceRequest may be recorded into current attempt
> ResourceRequests
> currentAttempt.updateResourceRequests(ask) ->
> // RM may allocate wrong AM Container for the current attempt, because its
> ResourceRequests
> // may come from previous attempt which can be any ResourceRequests previous
> AM asked
> // and there is not matching logic for the original AM Container
> ResourceRequest and
> // the returned amContainerAllocation below.
> AMContainerAllocatedTransition.transition(...) ->
> amContainerAllocation = scheduler.allocate(currentAttemptId, ...)
> {code}
> *Patch Correctness:*
> Because after this Patch, RM will definitely record ResourceRequests from
> different attempt into different objects of
> SchedulerApplicationAttempt.AppSchedulingInfo.
> So, even if RM still record ResourceRequests from old attempt at any time,
> these ResourceRequests will be recorded in old AppSchedulingInfo object which
> will not impact current attempt's resource requests and allocation.
> *Concerns:*
> The getApplicationAttempt function in AbstractYarnScheduler is so confusing,
> we should better rename it to getCurrentApplicationAttempt. And reconsider
> whether there are any other bugs related to getApplicationAttempt.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6885) AllocationFileLoaderService.loadQueue() should use a switch statement in the main tag parsing loop instead of the if/else-if/...
[
https://issues.apache.org/jira/browse/YARN-6885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16117743#comment-16117743
]
Yu-Tang Lin commented on YARN-6885:
---
Thanks to Daniel for filling this form, I would like to take this task, thanks!
> AllocationFileLoaderService.loadQueue() should use a switch statement in the
> main tag parsing loop instead of the if/else-if/...
>
>
> Key: YARN-6885
> URL: https://issues.apache.org/jira/browse/YARN-6885
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: fairscheduler
>Affects Versions: 3.0.0-alpha4
>Reporter: Daniel Templeton
>Priority: Minor
> Labels: newbie
>
> {code} if ("minResources".equals(field.getTagName())) {
> String text = ((Text)field.getFirstChild()).getData().trim();
> Resource val =
> FairSchedulerConfiguration.parseResourceConfigValue(text);
> minQueueResources.put(queueName, val);
> } else if ("maxResources".equals(field.getTagName())) {
> ...{code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-6212) NodeManager metrics returning wrong negative values
[
https://issues.apache.org/jira/browse/YARN-6212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16116452#comment-16116452
]
Yang Wang edited comment on YARN-6212 at 8/8/17 2:25 AM:
-
Hi, [~miklos.szeg...@cloudera.com]
I'm afraid this JIRA is not a duplicate of YARN-3933.
The primary cause of negative values is that metrics do not recover properly
when NM restart.
*AllocatedContainers,ContainersLaunched,AllocatedGB,AvailableGB,AllocatedVCores,AvailableVCores*
in metrics need to recover when NM restart.
This should be done in ContainerManagerImpl#recoverContainer.
The scenario could be reproduction by the following steps:
# Make sure
YarnConfiguration.NM_RECOVERY_ENABLED=true,YarnConfiguration.NM_RECOVERY_SUPERVISED=true
in NM
# Submit an application and keep running
# Restart NM
# Stop the application
# Now you get the negative values
was (Author: fly_in_gis):
Hi, Miklos Szegedi
I'm afraid this JIRA is not a duplicate of YARN-3933.
The primary cause of negative values is that metrics do not recover properly
when NM restart.
*AllocatedContainers,ContainersLaunched,AllocatedGB,AvailableGB,AllocatedVCores,AvailableVCores*
in metrics need to recover when NM restart.
This should be done in ContainerManagerImpl#recoverContainer.
The scenario could be reproduction by the following steps:
# Make sure
YarnConfiguration.NM_RECOVERY_ENABLED=true,YarnConfiguration.NM_RECOVERY_SUPERVISED=true
in NM
# Submit an application and keep running
# Restart NM
# Stop the application
# Now you get the negative values
> NodeManager metrics returning wrong negative values
> ---
>
> Key: YARN-6212
> URL: https://issues.apache.org/jira/browse/YARN-6212
> Project: Hadoop YARN
> Issue Type: Bug
> Components: metrics
>Affects Versions: 2.7.3
>Reporter: Abhishek Shivanna
>
> It looks like the metrics returned by the NodeManager have negative values
> for metrics that never should be negative. Here is an output form NM endpoint
> {noformat}
> /jmx?qry=Hadoop:service=NodeManager,name=NodeManagerMetrics
> {noformat}
> {noformat}
> {
> "beans" : [ {
> "name" : "Hadoop:service=NodeManager,name=NodeManagerMetrics",
> "modelerType" : "NodeManagerMetrics",
> "tag.Context" : "yarn",
> "tag.Hostname" : "",
> "ContainersLaunched" : 707,
> "ContainersCompleted" : 9,
> "ContainersFailed" : 124,
> "ContainersKilled" : 579,
> "ContainersIniting" : 0,
> "ContainersRunning" : 19,
> "AllocatedGB" : -26,
> "AllocatedContainers" : -5,
> "AvailableGB" : 252,
> "AllocatedVCores" : -5,
> "AvailableVCores" : 101,
> "ContainerLaunchDurationNumOps" : 718,
> "ContainerLaunchDurationAvgTime" : 18.0
> } ]
> }
> {noformat}
> Is there any circumstance under which the value for AllocatedGB,
> AllocatedContainers and AllocatedVCores go below 0?
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6940) FairScheduler: Enable Container update CodePaths and container resize testcase
[
https://issues.apache.org/jira/browse/YARN-6940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16117732#comment-16117732
]
Arun Suresh commented on YARN-6940:
---
ping [~templedf] / [~kasha] ?
> FairScheduler: Enable Container update CodePaths and container resize testcase
> --
>
> Key: YARN-6940
> URL: https://issues.apache.org/jira/browse/YARN-6940
> Project: Hadoop YARN
> Issue Type: Bug
> Components: fairscheduler
>Reporter: Arun Suresh
>Assignee: Arun Suresh
> Attachments: YARN-6940.001.patch
>
>
> After YARN-6216, the Container Update (which includes Resource increase and
> decrease) code-paths are mostly scheduler agnostic.
> This JIRA tracks the final minor change needed in the FairScheduler. It also
> re-enables the {{TestAMRMClient#testAMRMClientWithContainerResourceChange}}
> test for the FairScheduler - which verifies that it works for the
> FairScheduler.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6940) FairScheduler: Enable Container update CodePaths and container resize testcase
[
https://issues.apache.org/jira/browse/YARN-6940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun Suresh updated YARN-6940:
--
Summary: FairScheduler: Enable Container update CodePaths and container
resize testcase (was: FairScheduler: Enable Container update CodePath and
container resize testcase)
> FairScheduler: Enable Container update CodePaths and container resize testcase
> --
>
> Key: YARN-6940
> URL: https://issues.apache.org/jira/browse/YARN-6940
> Project: Hadoop YARN
> Issue Type: Bug
> Components: fairscheduler
>Reporter: Arun Suresh
>Assignee: Arun Suresh
> Attachments: YARN-6940.001.patch
>
>
> After YARN-6216, the Container Update (which includes Resource increase and
> decrease) code-paths are mostly scheduler agnostic.
> This JIRA tracks the final minor change needed in the FairScheduler. It also
> re-enables the {{TestAMRMClient#testAMRMClientWithContainerResourceChange}}
> test for the FairScheduler - which verifies that it works for the
> FairScheduler.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6940) FairScheduler: Enable Container update CodePath and container resize testcase
[
https://issues.apache.org/jira/browse/YARN-6940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun Suresh updated YARN-6940:
--
Summary: FairScheduler: Enable Container update CodePath and container
resize testcase (was: Enable Container Resize testcase for FairScheduler)
> FairScheduler: Enable Container update CodePath and container resize testcase
> -
>
> Key: YARN-6940
> URL: https://issues.apache.org/jira/browse/YARN-6940
> Project: Hadoop YARN
> Issue Type: Bug
> Components: fairscheduler
>Reporter: Arun Suresh
>Assignee: Arun Suresh
> Attachments: YARN-6940.001.patch
>
>
> After YARN-6216, the Container Update (which includes Resource increase and
> decrease) code-paths are mostly scheduler agnostic.
> This JIRA tracks the final minor change needed in the FairScheduler. It also
> re-enables the {{TestAMRMClient#testAMRMClientWithContainerResourceChange}}
> test for the FairScheduler - which verifies that it works for the
> FairScheduler.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6920) Fix resource leak that happens during container re-initialization.
[
https://issues.apache.org/jira/browse/YARN-6920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun Suresh updated YARN-6920:
--
Fix Version/s: 3.0.0-beta1
2.9.0
> Fix resource leak that happens during container re-initialization.
> --
>
> Key: YARN-6920
> URL: https://issues.apache.org/jira/browse/YARN-6920
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
>Reporter: Arun Suresh
>Assignee: Arun Suresh
> Fix For: 2.9.0, 3.0.0-beta1
>
> Attachments: YARN-6920.001.patch, YARN-6920.002.patch,
> YARN-6920.003.patch, YARN-6920.004.patch
>
>
> Looks like {{TestNMClient}} has been failing for a while. Opening this JIRA
> to track the fix.
> Upon Investigation:
> Container resources
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6920) Fix resource leak that happens during container re-initialization.
[
https://issues.apache.org/jira/browse/YARN-6920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun Suresh updated YARN-6920:
--
Component/s: nodemanager
> Fix resource leak that happens during container re-initialization.
> --
>
> Key: YARN-6920
> URL: https://issues.apache.org/jira/browse/YARN-6920
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
>Reporter: Arun Suresh
>Assignee: Arun Suresh
> Attachments: YARN-6920.001.patch, YARN-6920.002.patch,
> YARN-6920.003.patch, YARN-6920.004.patch
>
>
> Looks like {{TestNMClient}} has been failing for a while. Opening this JIRA
> to track the fix.
> Upon Investigation:
> Container resources
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6920) Fix resource leak that happens during container re-initialization.
[
https://issues.apache.org/jira/browse/YARN-6920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun Suresh updated YARN-6920:
--
Summary: Fix resource leak that happens during container re-initialization.
(was: Fix TestNMClient failure due to YARN-6706)
> Fix resource leak that happens during container re-initialization.
> --
>
> Key: YARN-6920
> URL: https://issues.apache.org/jira/browse/YARN-6920
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
>Reporter: Arun Suresh
>Assignee: Arun Suresh
> Attachments: YARN-6920.001.patch, YARN-6920.002.patch,
> YARN-6920.003.patch, YARN-6920.004.patch
>
>
> Looks like {{TestNMClient}} has been failing for a while. Opening this JIRA
> to track the fix.
> Upon Investigation:
> Container resources
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6920) Fix TestNMClient failure due to YARN-6706
[
https://issues.apache.org/jira/browse/YARN-6920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun Suresh updated YARN-6920:
--
Description:
Looks like {{TestNMClient}} has been failing for a while. Opening this JIRA to
track the fix.
Upon Investigation:
Container resources
was:
Looks like {{TestNMClient}} has been failing for a while. Opening this JIRA to
track the fix.
> Fix TestNMClient failure due to YARN-6706
> -
>
> Key: YARN-6920
> URL: https://issues.apache.org/jira/browse/YARN-6920
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Arun Suresh
>Assignee: Arun Suresh
> Attachments: YARN-6920.001.patch, YARN-6920.002.patch,
> YARN-6920.003.patch, YARN-6920.004.patch
>
>
> Looks like {{TestNMClient}} has been failing for a while. Opening this JIRA
> to track the fix.
> Upon Investigation:
> Container resources
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6668) Use cgroup to get container resource utilization
[
https://issues.apache.org/jira/browse/YARN-6668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16117708#comment-16117708
]
Hadoop QA commented on YARN-6668:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} YARN-1011 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
18s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 12m
49s{color} | {color:green} YARN-1011 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
46s{color} | {color:green} YARN-1011 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
50s{color} | {color:green} YARN-1011 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
12s{color} | {color:green} YARN-1011 passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
6s{color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common in
YARN-1011 has 1 extant Findbugs warnings. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
47s{color} | {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
in YARN-1011 has 5 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
3s{color} | {color:green} YARN-1011 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
12s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m
28s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 50s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch
generated 3 new + 16 unchanged - 1 fixed = 19 total (was 17) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
44s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 13m
52s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
34s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 65m 10s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | YARN-6668 |
| GITHUB PR | https://github.com/apache/hadoop/pull/241 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 582307be0df3 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12
13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | YARN-1011 / 4c501b4 |
| Default Java | 1.8.0_131 |
| findbugs | v3.1.0-RC1 |
| findbugs |
https://builds.apache.org/job/PreCommit-YARN-Build/16761/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common-warnings.html
|
| findbugs |
[jira] [Commented] (YARN-6852) [YARN-6223] Native code changes to support isolate GPU devices by using CGroups
[ https://issues.apache.org/jira/browse/YARN-6852?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16117702#comment-16117702 ] Miklos Szegedi commented on YARN-6852: -- No comments on the design, thanks for the explanation. We can keep container-executor but I would not add more - especially not converting existing files, since the standard is _. > [YARN-6223] Native code changes to support isolate GPU devices by using > CGroups > --- > > Key: YARN-6852 > URL: https://issues.apache.org/jira/browse/YARN-6852 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan >Assignee: Wangda Tan > Attachments: YARN-6852.001.patch, YARN-6852.002.patch, > YARN-6852.003.patch, YARN-6852.004.patch > > > This JIRA plan to add support of: > 1) Isolation in CGroups. (native side). -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5536) Multiple format support (JSON, etc.) for exclude node file in NM graceful decommission with timeout
[ https://issues-test.apache.org/jira/browse/YARN-5536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16090129#comment-16090129 ] Junping Du commented on YARN-5536: -- Oh. I see your point. Yes. If we can make this JIRA happen before we release YARN-4676, then we can safely drop current format with timeout. Let's bump up to critical (not blocker) given no one commit the bandwidth for JSON format yet. > Multiple format support (JSON, etc.) for exclude node file in NM graceful > decommission with timeout > --- > > Key: YARN-5536 > URL: https://issues-test.apache.org/jira/browse/YARN-5536 > Project: Hadoop YARN > Issue Type: Sub-task > Components: graceful >Reporter: Junping Du > > Per discussion in YARN-4676, we agree that multiple format (other than xml) > should be supported to decommission nodes with timeout values. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5536) Multiple format support (JSON, etc.) for exclude node file in NM graceful decommission with timeout
[ https://issues-test.apache.org/jira/browse/YARN-5536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du updated YARN-5536: - Priority: Critical (was: Major) > Multiple format support (JSON, etc.) for exclude node file in NM graceful > decommission with timeout > --- > > Key: YARN-5536 > URL: https://issues-test.apache.org/jira/browse/YARN-5536 > Project: Hadoop YARN > Issue Type: Sub-task > Components: graceful >Reporter: Junping Du >Priority: Critical > > Per discussion in YARN-4676, we agree that multiple format (other than xml) > should be supported to decommission nodes with timeout values. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6964) Fair scheduler misuses Resources operations
[
https://issues.apache.org/jira/browse/YARN-6964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16117694#comment-16117694
]
Miklos Szegedi commented on YARN-6964:
--
Thanks for the patch [~templedf].
{code}
566 Resource desiredShare =
567 Resources.componentwiseMin(getMinShare(), getDemand());
{code}
I am wondering, if this is the right thing to do. Let's say I have a container
that can run 6 threads parallel and it needs a 1GB buffer per thread. Now I
request 6 vcores and 6 GB but my min share is just 3 GB and 6 vcores. The
cluster has 1000GB and 1000 vcores. I will get a container with 6 vcores, 3 of
which I cannot use. The "dominant" in terms of min share is 3GB, so in case of
DRF, I should get 3GB and 3 vcores.
> Fair scheduler misuses Resources operations
> ---
>
> Key: YARN-6964
> URL: https://issues.apache.org/jira/browse/YARN-6964
> Project: Hadoop YARN
> Issue Type: Bug
> Components: fairscheduler
>Affects Versions: 3.0.0-alpha4
>Reporter: Daniel Templeton
>Assignee: Daniel Templeton
> Attachments: YARN-6964.001.patch
>
>
> There are several places where YARN uses the {{Resources}} class to do
> comparisons of {{Resource}} instances incorrectly. This patch corrects those
> mistakes.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6890) If UI is not secured, we allow user to kill other users' job even yarn cluster is secured.
[ https://issues.apache.org/jira/browse/YARN-6890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16117678#comment-16117678 ] Junping Du commented on YARN-6890: -- Thanks for review, Jian! v3 patch should incorporate your comments. > If UI is not secured, we allow user to kill other users' job even yarn > cluster is secured. > -- > > Key: YARN-6890 > URL: https://issues.apache.org/jira/browse/YARN-6890 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Sumana Sathish >Assignee: Junping Du >Priority: Critical > Attachments: YARN-6890.patch, YARN-6890-v2.patch, YARN-6890-v3.patch > > > Configuring SPNEGO for web browser could be a head ache, so many production > cluster choose to configure a unsecured UI access even for a secured cluster. > In this setup, users (login as some random guy) could watch other users job > which is expected. However, the kill button (added in YARN-3249 which enabled > by default) shouldn't work in this situation. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6890) If UI is not secured, we allow user to kill other users' job even yarn cluster is secured.
[ https://issues.apache.org/jira/browse/YARN-6890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du updated YARN-6890: - Attachment: YARN-6890-v3.patch > If UI is not secured, we allow user to kill other users' job even yarn > cluster is secured. > -- > > Key: YARN-6890 > URL: https://issues.apache.org/jira/browse/YARN-6890 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Sumana Sathish >Assignee: Junping Du >Priority: Critical > Attachments: YARN-6890.patch, YARN-6890-v2.patch, YARN-6890-v3.patch > > > Configuring SPNEGO for web browser could be a head ache, so many production > cluster choose to configure a unsecured UI access even for a secured cluster. > In this setup, users (login as some random guy) could watch other users job > which is expected. However, the kill button (added in YARN-3249 which enabled > by default) shouldn't work in this situation. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6033) Add support for sections in container-executor configuration file
[
https://issues.apache.org/jira/browse/YARN-6033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16117677#comment-16117677
]
Hadoop QA commented on YARN-6033:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
15s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 7 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
7s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
15s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 0m
37s{color} | {color:green}
hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager
generated 0 new + 0 unchanged - 1 fixed = 0 total (was 1) {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 14m
1s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 31m 5s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | YARN-6033 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12880750/YARN-6033.012.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit xml cc |
| uname | Linux 5476395dfbed 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12
13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / c61f2c4 |
| Default Java | 1.8.0_131 |
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/16760/testReport/ |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/16760/console |
| Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Add support for sections in container-executor configuration file
> -
>
> Key: YARN-6033
> URL: https://issues.apache.org/jira/browse/YARN-6033
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager
>Reporter: Varun Vasudev
>Assignee: Varun Vasudev
> Attachments: YARN-6033.003.patch, YARN-6033.004.patch,
> YARN-6033.005.patch, YARN-6033.006.patch, YARN-6033.007.patch,
> YARN-6033.008.patch, YARN-6033.009.patch, YARN-6033.010.patch,
> YARN-6033.011.patch, YARN-6033.012.patch, YARN-6033-YARN-5673.001.patch,
> YARN-6033-YARN-5673.002.patch
>
>
--
This message
[jira] [Commented] (YARN-5536) Multiple format support (JSON, etc.) for exclude node file in NM graceful decommission with timeout
[ https://issues-test.apache.org/jira/browse/YARN-5536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16090128#comment-16090128 ] Ming Ma commented on YARN-5536: --- I am not suggesting removing the previous format support (without timeout). We need that as it is used by everyone to decommission nodes. But adding the timeout support to the previous format by YARN-4676 isn't my preference. Given YARN-4676 hasn't been released yet, we can still afford to remove the timeout support from the previous format without breaking backward compatibility. That means if people want to use graceful decommission with timeout, they have to use the json format. I think that is an acceptable requirement. > Multiple format support (JSON, etc.) for exclude node file in NM graceful > decommission with timeout > --- > > Key: YARN-5536 > URL: https://issues-test.apache.org/jira/browse/YARN-5536 > Project: Hadoop YARN > Issue Type: Sub-task > Components: graceful >Reporter: Junping Du > > Per discussion in YARN-4676, we agree that multiple format (other than xml) > should be supported to decommission nodes with timeout values. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6033) Add support for sections in container-executor configuration file
[
https://issues.apache.org/jira/browse/YARN-6033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16117675#comment-16117675
]
Miklos Szegedi commented on YARN-6033:
--
Thank you for the patch [~wangda]. The last comments are style only, the patch
looks good to me otherwise (non-binding).
{code}
493 cfg->sections = (struct section **) malloc(
494 sizeof(struct section *) * MAX_SIZE);
{code}
There is a missing NULL check here.
{code}
466 if (free_second_section) {
467 free(section2->name);
468 memset(section2, 0, sizeof(*section2));
469 free(section2);
470 }
{code}
I commented earlier to remove free_second_section. If we keep it, we might want
to add a unit test and still do memset(section2, 0, sizeof(*section2)), if
free_second_section is 0; This is a note only. We can do it in another jira,
since this is dead code.
{code}
513 const struct section *existing_section =
514 get_configuration_section(new_section->name, cfg);
{code}
This variable is actually not const. We modify the structure.
> Add support for sections in container-executor configuration file
> -
>
> Key: YARN-6033
> URL: https://issues.apache.org/jira/browse/YARN-6033
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager
>Reporter: Varun Vasudev
>Assignee: Varun Vasudev
> Attachments: YARN-6033.003.patch, YARN-6033.004.patch,
> YARN-6033.005.patch, YARN-6033.006.patch, YARN-6033.007.patch,
> YARN-6033.008.patch, YARN-6033.009.patch, YARN-6033.010.patch,
> YARN-6033.011.patch, YARN-6033.012.patch, YARN-6033-YARN-5673.001.patch,
> YARN-6033-YARN-5673.002.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6920) Fix TestNMClient failure due to YARN-6706
[
https://issues.apache.org/jira/browse/YARN-6920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16117670#comment-16117670
]
Hadoop QA commented on YARN-6920:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m
3s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
59s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
5s{color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
53s{color} | {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
in trunk has 5 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
48s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
12s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 5m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 13m
32s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 19m
45s{color} | {color:green} hadoop-yarn-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
27s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 82m 14s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | YARN-6920 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12880112/YARN-6920.004.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux e676d3b40006 3.13.0-119-generic #166-Ubuntu SMP Wed May 3
12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / bc20680 |
| Default Java | 1.8.0_131 |
| findbugs | v3.1.0-RC1 |
| findbugs |
https://builds.apache.org/job/PreCommit-YARN-Build/16758/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-warnings.html
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/16758/testReport/ |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client U:
hadoop-yarn-project/hadoop-yarn |
| Console output |
[jira] [Commented] (YARN-5536) Multiple format support (JSON, etc.) for exclude node file in NM graceful decommission with timeout
[ https://issues-test.apache.org/jira/browse/YARN-5536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16090127#comment-16090127 ] Junping Du commented on YARN-5536: -- Hi [~mingma], I am more conservative on removing previous format support given original format (without timeout) has been used for years for cluster management (configuration compatibility should be treated as API). If we all agree JSON format is the right direction we are heading to, the next step should be deprecate it in next major release (3.x), and then totally drop it since Hadoop 4.x. Make sense? > Multiple format support (JSON, etc.) for exclude node file in NM graceful > decommission with timeout > --- > > Key: YARN-5536 > URL: https://issues-test.apache.org/jira/browse/YARN-5536 > Project: Hadoop YARN > Issue Type: Sub-task > Components: graceful >Reporter: Junping Du > > Per discussion in YARN-4676, we agree that multiple format (other than xml) > should be supported to decommission nodes with timeout values. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6668) Use cgroup to get container resource utilization
[ https://issues.apache.org/jira/browse/YARN-6668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Szegedi updated YARN-6668: - Attachment: YARN-6668.004.patch > Use cgroup to get container resource utilization > > > Key: YARN-6668 > URL: https://issues.apache.org/jira/browse/YARN-6668 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Affects Versions: 3.0.0-alpha3 >Reporter: Haibo Chen >Assignee: Miklos Szegedi > Attachments: YARN-6668.000.patch, YARN-6668.001.patch, > YARN-6668.002.patch, YARN-6668.003.patch, YARN-6668.004.patch > > > Container Monitor relies on proc file system to get container resource > utilization, which is not as efficient as reading cgroup accounting. We > should in NM, when cgroup is enabled, read cgroup stats instead. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6964) Fair scheduler misuses Resources operations
[
https://issues.apache.org/jira/browse/YARN-6964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Templeton updated YARN-6964:
---
Attachment: YARN-6964.001.patch
> Fair scheduler misuses Resources operations
> ---
>
> Key: YARN-6964
> URL: https://issues.apache.org/jira/browse/YARN-6964
> Project: Hadoop YARN
> Issue Type: Bug
> Components: fairscheduler
>Affects Versions: 3.0.0-alpha4
>Reporter: Daniel Templeton
>Assignee: Daniel Templeton
> Attachments: YARN-6964.001.patch
>
>
> There are several places where YARN uses the {{Resources}} class to do
> comparisons of {{Resource}} instances incorrectly. This patch corrects those
> mistakes.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-6964) Fair scheduler misuses Resources operations
Daniel Templeton created YARN-6964:
--
Summary: Fair scheduler misuses Resources operations
Key: YARN-6964
URL: https://issues.apache.org/jira/browse/YARN-6964
Project: Hadoop YARN
Issue Type: Bug
Components: fairscheduler
Affects Versions: 3.0.0-alpha4
Reporter: Daniel Templeton
Assignee: Daniel Templeton
There are several places where YARN uses the {{Resources}} class to do
comparisons of {{Resource}} instances incorrectly. This patch corrects those
mistakes.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6033) Add support for sections in container-executor configuration file
[
https://issues.apache.org/jira/browse/YARN-6033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wangda Tan updated YARN-6033:
-
Attachment: YARN-6033.012.patch
Attached ver.012 patch, should addressed all your comments.
bq. I already commented on this but now I have the proof how it can go wrong.
If populate_section_fields does not get any new entries, then cfg->sections is
malloced but cfg->size=0. That means that is this code will not free the memory:
Make sense, so the following fix should be enough, correct?
{code}
if (cfg->sections) {
free(cfg->sections);
cfg->sections = NULL;
}
{code}
> Add support for sections in container-executor configuration file
> -
>
> Key: YARN-6033
> URL: https://issues.apache.org/jira/browse/YARN-6033
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager
>Reporter: Varun Vasudev
>Assignee: Varun Vasudev
> Attachments: YARN-6033.003.patch, YARN-6033.004.patch,
> YARN-6033.005.patch, YARN-6033.006.patch, YARN-6033.007.patch,
> YARN-6033.008.patch, YARN-6033.009.patch, YARN-6033.010.patch,
> YARN-6033.011.patch, YARN-6033.012.patch, YARN-6033-YARN-5673.001.patch,
> YARN-6033-YARN-5673.002.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5536) Multiple format support (JSON, etc.) for exclude node file in NM graceful decommission with timeout
[ https://issues-test.apache.org/jira/browse/YARN-5536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16090126#comment-16090126 ] Ming Ma commented on YARN-5536: --- Per discussion in YARN-4676, current timeout config support is via adding another column to the host file. Using host file means we have to support such timeout config moving forward for backward compatibility. So I would prefer we remove such support before we release 2.9 and use json format instead. If so, this jira should be a blocker. [~djp], thoughts? > Multiple format support (JSON, etc.) for exclude node file in NM graceful > decommission with timeout > --- > > Key: YARN-5536 > URL: https://issues-test.apache.org/jira/browse/YARN-5536 > Project: Hadoop YARN > Issue Type: Sub-task > Components: graceful >Reporter: Junping Du > > Per discussion in YARN-4676, we agree that multiple format (other than xml) > should be supported to decommission nodes with timeout values. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Resolved] (YARN-1038) LocalizationProtocolPBClientImpl RPC failing
[
https://issues-test.apache.org/jira/browse/YARN-1038?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Junping Du resolved YARN-1038.
--
Resolution: Cannot Reproduce
I don't think trunk branch has this problem now, just resolve as cannot
reproduce.
> LocalizationProtocolPBClientImpl RPC failing
>
>
> Key: YARN-1038
> URL: https://issues-test.apache.org/jira/browse/YARN-1038
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 3.0.0-alpha1
>Reporter: Alejandro Abdelnur
>Priority: Blocker
>
> Trying to run an MR job in trunk is failing with:
> {code}
> 2013-08-06 22:24:21,498 WARN org.apache.hadoop.ipc.Client: interrupted
> waiting to send rpc request to server
> java.lang.InterruptedException
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1279)
> at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:218)
> at java.util.concurrent.FutureTask.get(FutureTask.java:83)
> at
> org.apache.hadoop.ipc.Client$Connection.sendRpcRequest(Client.java:1019)
> at org.apache.hadoop.ipc.Client.call(Client.java:1372)
> at org.apache.hadoop.ipc.Client.call(Client.java:1352)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
> at com.sun.proxy.$Proxy25.heartbeat(Unknown Source)
> at
> org.apache.hadoop.yarn.server.nodemanager.api.impl.pb.client.LocalizationProtocolPBClientImpl.heartbeat(LocalizationProtocolPBClientImpl.java:62)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer.localizeFiles(ContainerLocalizer.java:250)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer.runLocalization(ContainerLocalizer.java:164)
> at
> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.startLocalizer(DefaultContainerExecutor.java:107)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:977)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-6852) [YARN-6223] Native code changes to support isolate GPU devices by using CGroups
[ https://issues.apache.org/jira/browse/YARN-6852?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16117607#comment-16117607 ] Wangda Tan edited comment on YARN-6852 at 8/8/17 12:19 AM: --- Again, thanks [~miklos.szeg...@cloudera.com] for your reviews. bq. could not find a doc about this, so I ask. Is there a reason to have a disable list of devices, instead of an enable list? Actually whitelist doesn't work for me, I tried to debug however I cannot find the root cause. Since blacklist and whitelist are just two different approaches for the same purpose. I think we could continue this approach, do you have any concern here? And regarding to file name style. I was just about to rename to underscore, however I found the container-executor itself is "\-", I suggest to convert existing files to "\-" connected, otherwise we have to rename a lot of files. I don't quite care about which style to go, I just want to avoid change too many file. Attached ver.004 patch, (TODO): unit test related cleanups. was (Author: leftnoteasy): Again, thanks [~miklos.szeg...@cloudera.com] for your reviews. bq. could not find a doc about this, so I ask. Is there a reason to have a disable list of devices, instead of an enable list? Actually whitelist doesn't work for me, I tried to debug however I cannot find the root cause. Since blacklist and whitelist are just two different approaches for the same purpose. I think we could continue this approach, do you have any concern here? And regarding to file name style. I was just about to rename to underscore, however I found the container-executor itself is "-", I suggest to convert existing files to "-" connected, otherwise we have to rename a lot of files. I don't quite care about which style to go, I just want to avoid change too many file. Attached ver.004 patch, (TODO): unit test related cleanups. > [YARN-6223] Native code changes to support isolate GPU devices by using > CGroups > --- > > Key: YARN-6852 > URL: https://issues.apache.org/jira/browse/YARN-6852 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan >Assignee: Wangda Tan > Attachments: YARN-6852.001.patch, YARN-6852.002.patch, > YARN-6852.003.patch, YARN-6852.004.patch > > > This JIRA plan to add support of: > 1) Isolation in CGroups. (native side). -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6852) [YARN-6223] Native code changes to support isolate GPU devices by using CGroups
[ https://issues.apache.org/jira/browse/YARN-6852?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-6852: - Attachment: YARN-6852.004.patch Again, thanks [~miklos.szeg...@cloudera.com] for your reviews. bq. could not find a doc about this, so I ask. Is there a reason to have a disable list of devices, instead of an enable list? Actually whitelist doesn't work for me, I tried to debug however I cannot find the root cause. Since blacklist and whitelist are just two different approaches for the same purpose. I think we could continue this approach, do you have any concern here? And regarding to file name style. I was just about to rename to underscore, however I found the container-executor itself is "-", I suggest to convert existing files to "-" connected, otherwise we have to rename a lot of files. I don't quite care about which style to go, I just want to avoid change too many file. Attached ver.004 patch, (TODO): unit test related cleanups. > [YARN-6223] Native code changes to support isolate GPU devices by using > CGroups > --- > > Key: YARN-6852 > URL: https://issues.apache.org/jira/browse/YARN-6852 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan >Assignee: Wangda Tan > Attachments: YARN-6852.001.patch, YARN-6852.002.patch, > YARN-6852.003.patch, YARN-6852.004.patch > > > This JIRA plan to add support of: > 1) Isolation in CGroups. (native side). -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6033) Add support for sections in container-executor configuration file
[
https://issues.apache.org/jira/browse/YARN-6033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16117604#comment-16117604
]
Miklos Szegedi commented on YARN-6033:
--
{code}
485 cfg->size = 0;
486 conf_file = fopen(file_path, "r");
487 if (conf_file == NULL) {
488fprintf(ERRORFILE, "Invalid conf file provided, unable to open file"
489" : %s \n", file_path);
490return (INVALID_CONFIG_FILE);
491 }
492
493 cfg->sections = (struct section **) malloc(
494sizeof(struct section *) * MAX_SIZE);
{code}
I already commented on this but now I have the proof how it can go wrong. If
populate_section_fields does not get any new entries, then cfg->sections is
malloced but cfg->size=0. That means that is this code will not free the memory:
{code}
70 if (cfg->size > 0) {
71free(cfg->sections);
72 }
{code}
It should rather be:
{code}
if (cfg->sections) {
free(cfg->sections);
cfg->sections = NULL;
}
{code}
-
{code}
36return_values = (char **) malloc(sizeof(char *) * return_values_size);
37memset(return_values, 0, sizeof(char *) * return_values_size);
{code}
There is no NULL check here.
{code}
50 // Make sure returned values has enough space for the trailing NULL.
51 if (size >= return_values_size - 2) {
{code}
The code needs to make sure that there is one item free, either for NULL, or
for the next item. So I do not think that you need the -2 here.
> Add support for sections in container-executor configuration file
> -
>
> Key: YARN-6033
> URL: https://issues.apache.org/jira/browse/YARN-6033
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager
>Reporter: Varun Vasudev
>Assignee: Varun Vasudev
> Attachments: YARN-6033.003.patch, YARN-6033.004.patch,
> YARN-6033.005.patch, YARN-6033.006.patch, YARN-6033.007.patch,
> YARN-6033.008.patch, YARN-6033.009.patch, YARN-6033.010.patch,
> YARN-6033.011.patch, YARN-6033-YARN-5673.001.patch,
> YARN-6033-YARN-5673.002.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-6910) Increase RM audit log coverage
[
https://issues.apache.org/jira/browse/YARN-6910?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhenzhao wang reassigned YARN-6910:
---
Assignee: zhenzhao wang
> Increase RM audit log coverage
> --
>
> Key: YARN-6910
> URL: https://issues.apache.org/jira/browse/YARN-6910
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: Ming Ma
>Assignee: zhenzhao wang
>
> RM's audit logger logs certain API calls. It will be useful to increase its
> coverage to include methods like {{getApplications}}. It addition, the audit
> logger should track calls from rest APIs in addition to RPC calls.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6955) Handle concurrent register AM requests in FederationInterceptor
[ https://issues.apache.org/jira/browse/YARN-6955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16117596#comment-16117596 ] Botong Huang commented on YARN-6955: Thanks [~subru]! > Handle concurrent register AM requests in FederationInterceptor > --- > > Key: YARN-6955 > URL: https://issues.apache.org/jira/browse/YARN-6955 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Botong Huang >Assignee: Botong Huang >Priority: Minor > Fix For: 3.0.0-beta1 > > Attachments: YARN-6955.v1.patch, YARN-6955.v2.patch > > > The timeout between AM and AMRMProxy is shorter than the timeout + failOver > between FederationInterceptor (AMRMProxy) and RM. When the first register > thread in FI is blocked because of an RM failover, AM can timeout and resend > register call, leading to two outstanding register call inside FI. > Eventually when RM comes back up, one thread succeeds register and the other > thread got an application already registered exception. FI should swallow the > exception and return success back to AM in both threads. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-1038) LocalizationProtocolPBClientImpl RPC failing
[
https://issues-test.apache.org/jira/browse/YARN-1038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16090124#comment-16090124
]
Ming Ma commented on YARN-1038:
---
Given this blocker was opened many years ago and there isn't any recent
complaints like this, seems we should just close it.
> LocalizationProtocolPBClientImpl RPC failing
>
>
> Key: YARN-1038
> URL: https://issues-test.apache.org/jira/browse/YARN-1038
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 3.0.0-alpha1
>Reporter: Alejandro Abdelnur
>Priority: Blocker
>
> Trying to run an MR job in trunk is failing with:
> {code}
> 2013-08-06 22:24:21,498 WARN org.apache.hadoop.ipc.Client: interrupted
> waiting to send rpc request to server
> java.lang.InterruptedException
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1279)
> at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:218)
> at java.util.concurrent.FutureTask.get(FutureTask.java:83)
> at
> org.apache.hadoop.ipc.Client$Connection.sendRpcRequest(Client.java:1019)
> at org.apache.hadoop.ipc.Client.call(Client.java:1372)
> at org.apache.hadoop.ipc.Client.call(Client.java:1352)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
> at com.sun.proxy.$Proxy25.heartbeat(Unknown Source)
> at
> org.apache.hadoop.yarn.server.nodemanager.api.impl.pb.client.LocalizationProtocolPBClientImpl.heartbeat(LocalizationProtocolPBClientImpl.java:62)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer.localizeFiles(ContainerLocalizer.java:250)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer.runLocalization(ContainerLocalizer.java:164)
> at
> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.startLocalizer(DefaultContainerExecutor.java:107)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:977)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6668) Use cgroup to get container resource utilization
[
https://issues.apache.org/jira/browse/YARN-6668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16117587#comment-16117587
]
Haibo Chen commented on YARN-6668:
--
Two more comments
1) all the statements, "FileUtils.deleteDirectory(new File("/tmp/" +
this.getClass().getName()));" are not reference the new field procfsBaseDir.
2) Instead of using @Rule, we could do something @Test (expected =
YarnException.class) .
> Use cgroup to get container resource utilization
>
>
> Key: YARN-6668
> URL: https://issues.apache.org/jira/browse/YARN-6668
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager
>Affects Versions: 3.0.0-alpha3
>Reporter: Haibo Chen
>Assignee: Miklos Szegedi
> Attachments: YARN-6668.000.patch, YARN-6668.001.patch,
> YARN-6668.002.patch, YARN-6668.003.patch
>
>
> Container Monitor relies on proc file system to get container resource
> utilization, which is not as efficient as reading cgroup accounting. We
> should in NM, when cgroup is enabled, read cgroup stats instead.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6668) Use cgroup to get container resource utilization
[
https://issues.apache.org/jira/browse/YARN-6668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16117585#comment-16117585
]
Hadoop QA commented on YARN-6668:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} YARN-1011 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
24s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
26s{color} | {color:green} YARN-1011 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
33s{color} | {color:green} YARN-1011 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
53s{color} | {color:green} YARN-1011 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
12s{color} | {color:green} YARN-1011 passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
5s{color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common in
YARN-1011 has 1 extant Findbugs warnings. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
53s{color} | {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
in YARN-1011 has 5 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
0s{color} | {color:green} YARN-1011 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m
38s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 50s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch
generated 3 new + 16 unchanged - 1 fixed = 19 total (was 17) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
40s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 13m
28s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
32s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 70m 0s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | YARN-6668 |
| GITHUB PR | https://github.com/apache/hadoop/pull/241 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 3ae551539850 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31
14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | YARN-1011 / 4c501b4 |
| Default Java | 1.8.0_131 |
| findbugs | v3.1.0-RC1 |
| findbugs |
https://builds.apache.org/job/PreCommit-YARN-Build/16757/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common-warnings.html
|
| findbugs |
[jira] [Updated] (YARN-6903) Yarn-native-service framework core rewrite
[ https://issues.apache.org/jira/browse/YARN-6903?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jian He updated YARN-6903: -- Attachment: YARN-6903.yarn-native-services.03.patch > Yarn-native-service framework core rewrite > -- > > Key: YARN-6903 > URL: https://issues.apache.org/jira/browse/YARN-6903 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jian He >Assignee: Jian He > Attachments: YARN-6903.yarn-native-services.01.patch, > YARN-6903.yarn-native-services.02.patch, > YARN-6903.yarn-native-services.03.patch > > > There are some new features like rich placement scheduling, container auto > restart, container upgrade in YARN core that can be taken advantage by the > native-service framework. Besides, there are quite a lot legacy code which > are no longer required. > So we decide to rewrite the core part to have a leaner codebase and make use > of various advanced features in YARN. > And the new code design will be in align with what we have designed for the > service API YARN-4793 -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6917) Queue path is recomputed from scratch on every allocation
[
https://issues.apache.org/jira/browse/YARN-6917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16117563#comment-16117563
]
Hadoop QA commented on YARN-6917:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 2m
43s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
34s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
59s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
21s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 24s{color} | {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 1 new + 97 unchanged - 0 fixed = 98 total (was 97) {color}
|
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 43m
34s{color} | {color:green} hadoop-yarn-server-resourcemanager in the patch
passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 68m 11s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | YARN-6917 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12880723/YARN-6917.001.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 1f538a0eee8b 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31
14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / adb84f3 |
| Default Java | 1.8.0_131 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/16756/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/16756/testReport/ |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/16756/console |
| Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Queue path is recomputed from scratch on every allocation
>
[jira] [Commented] (YARN-6962) Add support for updateContainers when allocating using FederationInterceptor
[ https://issues.apache.org/jira/browse/YARN-6962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16117553#comment-16117553 ] Subru Krishnan commented on YARN-6962: -- Thanks [~botong] for working on this. The patch itself LGTM, can you please add a simple test? > Add support for updateContainers when allocating using FederationInterceptor > > > Key: YARN-6962 > URL: https://issues.apache.org/jira/browse/YARN-6962 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Botong Huang >Assignee: Botong Huang >Priority: Minor > Attachments: YARN-6962.v1.patch > > > Container update is introduced in YARN-5221. Federation Interceptor needs to > support it when splitting (merging) the allocate request (response). -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6920) Fix TestNMClient failure due to YARN-6706
[
https://issues.apache.org/jira/browse/YARN-6920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16117547#comment-16117547
]
Hadoop QA commented on YARN-6920:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
29s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
12s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 13m
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
25s{color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
15s{color} | {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
in trunk has 5 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
10s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
14s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 8m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 15m 18s{color}
| {color:red} hadoop-yarn-server-nodemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 21m
15s{color} | {color:green} hadoop-yarn-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
33s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 99m 20s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.nodemanager.containermanager.TestContainerManager |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | YARN-6920 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12880112/YARN-6920.004.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux b7693b3f98b5 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31
14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / adb84f3 |
| Default Java | 1.8.0_131 |
| findbugs | v3.1.0-RC1 |
| findbugs |
https://builds.apache.org/job/PreCommit-YARN-Build/16752/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-warnings.html
|
| unit |
https://builds.apache.org/job/PreCommit-YARN-Build/16752/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
|
| Test
[jira] [Updated] (YARN-6962) Add support for updateContainers when allocating using FederationInterceptor
[ https://issues.apache.org/jira/browse/YARN-6962?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Botong Huang updated YARN-6962: --- Description: Container update is introduced in YARN-5221. Federation Interceptor needs to support it when splitting (merging) the allocate request (response). > Add support for updateContainers when allocating using FederationInterceptor > > > Key: YARN-6962 > URL: https://issues.apache.org/jira/browse/YARN-6962 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Botong Huang >Assignee: Botong Huang >Priority: Minor > Attachments: YARN-6962.v1.patch > > > Container update is introduced in YARN-5221. Federation Interceptor needs to > support it when splitting (merging) the allocate request (response). -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6962) Add support for updateContainers when allocating using FederationInterceptor
[ https://issues.apache.org/jira/browse/YARN-6962?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Botong Huang updated YARN-6962: --- Summary: Add support for updateContainers when allocating using FederationInterceptor (was: Federation interceptor should support full allocate request/response api) > Add support for updateContainers when allocating using FederationInterceptor > > > Key: YARN-6962 > URL: https://issues.apache.org/jira/browse/YARN-6962 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Botong Huang >Assignee: Botong Huang >Priority: Minor > Attachments: YARN-6962.v1.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6955) Handle concurrent register AM requests in FederationInterceptor
[ https://issues.apache.org/jira/browse/YARN-6955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Subru Krishnan updated YARN-6955: - Summary: Handle concurrent register AM requests in FederationInterceptor (was: Concurrent registerAM thread in Federation Interceptor) > Handle concurrent register AM requests in FederationInterceptor > --- > > Key: YARN-6955 > URL: https://issues.apache.org/jira/browse/YARN-6955 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Botong Huang >Assignee: Botong Huang >Priority: Minor > Attachments: YARN-6955.v1.patch, YARN-6955.v2.patch > > > The timeout between AM and AMRMProxy is shorter than the timeout + failOver > between FederationInterceptor (AMRMProxy) and RM. When the first register > thread in FI is blocked because of an RM failover, AM can timeout and resend > register call, leading to two outstanding register call inside FI. > Eventually when RM comes back up, one thread succeeds register and the other > thread got an application already registered exception. FI should swallow the > exception and return success back to AM in both threads. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6930) Admins should be able to explicitly enable specific LinuxContainerRuntime in the NodeManager
[
https://issues.apache.org/jira/browse/YARN-6930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16117508#comment-16117508
]
Hadoop QA commented on YARN-6930:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
1s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
28s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
54s{color} | {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
in trunk has 5 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
8s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 5m
51s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 4s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch
generated 2 new + 215 unchanged - 0 fixed = 217 total (was 215) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
18s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
23s{color} | {color:red}
hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager
generated 1 new + 105 unchanged - 0 fixed = 106 total (was 105) {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 33s{color}
| {color:red} hadoop-yarn-api in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 13m
26s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
14s{color} | {color:green} hadoop-yarn-site in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
26s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 66m 29s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.yarn.conf.TestYarnConfigurationFields |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | YARN-6930 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12880714/YARN-6930.001.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit
[jira] [Commented] (YARN-6890) If UI is not secured, we allow user to kill other users' job even yarn cluster is secured.
[
https://issues.apache.org/jira/browse/YARN-6890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16117505#comment-16117505
]
Jian He commented on YARN-6890:
---
- Looks good overall, minor comment, there's an
UserGroupInformation#isSecurityEnabled API for checking if the cluster is
secured, we can use that instead.
{code}
this.unsecuredUIForSecuredCluster = (auth != null) &&
auth.equals("kerberos") &&
(httpAuth != null) && httpAuth.equals("simple");
{code}
> If UI is not secured, we allow user to kill other users' job even yarn
> cluster is secured.
> --
>
> Key: YARN-6890
> URL: https://issues.apache.org/jira/browse/YARN-6890
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Sumana Sathish
>Assignee: Junping Du
>Priority: Critical
> Attachments: YARN-6890.patch, YARN-6890-v2.patch
>
>
> Configuring SPNEGO for web browser could be a head ache, so many production
> cluster choose to configure a unsecured UI access even for a secured cluster.
> In this setup, users (login as some random guy) could watch other users job
> which is expected. However, the kill button (added in YARN-3249 which enabled
> by default) shouldn't work in this situation.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6033) Add support for sections in container-executor configuration file
[
https://issues.apache.org/jira/browse/YARN-6033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16117495#comment-16117495
]
Hadoop QA commented on YARN-6033:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
14s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 7 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
52s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
18s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 0m
38s{color} | {color:green}
hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager
generated 0 new + 0 unchanged - 1 fixed = 0 total (was 1) {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 13m
18s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 31m 16s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | YARN-6033 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12880718/YARN-6033.011.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit xml cc |
| uname | Linux 084cff99810b 3.13.0-119-generic #166-Ubuntu SMP Wed May 3
12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / adb84f3 |
| Default Java | 1.8.0_131 |
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/16755/testReport/ |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/16755/console |
| Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Add support for sections in container-executor configuration file
> -
>
> Key: YARN-6033
> URL: https://issues.apache.org/jira/browse/YARN-6033
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager
>Reporter: Varun Vasudev
>Assignee: Varun Vasudev
> Attachments: YARN-6033.003.patch, YARN-6033.004.patch,
> YARN-6033.005.patch, YARN-6033.006.patch, YARN-6033.007.patch,
> YARN-6033.008.patch, YARN-6033.009.patch, YARN-6033.010.patch,
> YARN-6033.011.patch, YARN-6033-YARN-5673.001.patch,
> YARN-6033-YARN-5673.002.patch
>
>
--
This message was sent by Atlassian
[jira] [Updated] (YARN-6668) Use cgroup to get container resource utilization
[ https://issues.apache.org/jira/browse/YARN-6668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Szegedi updated YARN-6668: - Attachment: YARN-6668.003.patch Thank you for the comments [~haibochen], I updated the patch. > Use cgroup to get container resource utilization > > > Key: YARN-6668 > URL: https://issues.apache.org/jira/browse/YARN-6668 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Affects Versions: 3.0.0-alpha3 >Reporter: Haibo Chen >Assignee: Miklos Szegedi > Attachments: YARN-6668.000.patch, YARN-6668.001.patch, > YARN-6668.002.patch, YARN-6668.003.patch > > > Container Monitor relies on proc file system to get container resource > utilization, which is not as efficient as reading cgroup accounting. We > should in NM, when cgroup is enabled, read cgroup stats instead. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6668) Use cgroup to get container resource utilization
[ https://issues.apache.org/jira/browse/YARN-6668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16117467#comment-16117467 ] Miklos Szegedi commented on YARN-6668: -- bq. 3) The CGroupsResoureCalculator now takes precedence over user-configuration // CGroups is best in performance, so try to use it, if it is enabled > Use cgroup to get container resource utilization > > > Key: YARN-6668 > URL: https://issues.apache.org/jira/browse/YARN-6668 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Affects Versions: 3.0.0-alpha3 >Reporter: Haibo Chen >Assignee: Miklos Szegedi > Attachments: YARN-6668.000.patch, YARN-6668.001.patch, > YARN-6668.002.patch > > > Container Monitor relies on proc file system to get container resource > utilization, which is not as efficient as reading cgroup accounting. We > should in NM, when cgroup is enabled, read cgroup stats instead. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6668) Use cgroup to get container resource utilization
[ https://issues.apache.org/jira/browse/YARN-6668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16117464#comment-16117464 ] Miklos Szegedi commented on YARN-6668: -- bq. 5) ContainersMonitorImpl is missing an import of YarnException, which leads to compilation failure Hmm. I see it in my local copy. > Use cgroup to get container resource utilization > > > Key: YARN-6668 > URL: https://issues.apache.org/jira/browse/YARN-6668 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Affects Versions: 3.0.0-alpha3 >Reporter: Haibo Chen >Assignee: Miklos Szegedi > Attachments: YARN-6668.000.patch, YARN-6668.001.patch, > YARN-6668.002.patch > > > Container Monitor relies on proc file system to get container resource > utilization, which is not as efficient as reading cgroup accounting. We > should in NM, when cgroup is enabled, read cgroup stats instead. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6668) Use cgroup to get container resource utilization
[ https://issues.apache.org/jira/browse/YARN-6668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16117454#comment-16117454 ] Miklos Szegedi commented on YARN-6668: -- bq. 9) We call setCGroupFilePaths() in constructor which reads file. Can we move that out into an init() method instead? It just allocates File objects, they are not actually used, yet. They are more like a path. > Use cgroup to get container resource utilization > > > Key: YARN-6668 > URL: https://issues.apache.org/jira/browse/YARN-6668 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Affects Versions: 3.0.0-alpha3 >Reporter: Haibo Chen >Assignee: Miklos Szegedi > Attachments: YARN-6668.000.patch, YARN-6668.001.patch, > YARN-6668.002.patch > > > Container Monitor relies on proc file system to get container resource > utilization, which is not as efficient as reading cgroup accounting. We > should in NM, when cgroup is enabled, read cgroup stats instead. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6668) Use cgroup to get container resource utilization
[ https://issues.apache.org/jira/browse/YARN-6668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16117445#comment-16117445 ] Miklos Szegedi commented on YARN-6668: -- bq. the lock around firstError is unnecessary if we make firstError volatile This is not enough actually, since the whole test and set + print has to be protected, otherwise we may print out multiple times. > Use cgroup to get container resource utilization > > > Key: YARN-6668 > URL: https://issues.apache.org/jira/browse/YARN-6668 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Affects Versions: 3.0.0-alpha3 >Reporter: Haibo Chen >Assignee: Miklos Szegedi > Attachments: YARN-6668.000.patch, YARN-6668.001.patch, > YARN-6668.002.patch > > > Container Monitor relies on proc file system to get container resource > utilization, which is not as efficient as reading cgroup accounting. We > should in NM, when cgroup is enabled, read cgroup stats instead. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6917) Queue path is recomputed from scratch on every allocation
[
https://issues.apache.org/jira/browse/YARN-6917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Payne updated YARN-6917:
-
Attachment: YARN-6917.001.patch
The queue path is only ever set during queue initialization or
reinitialization. During this time, new queue objects are created and set into
a new hierarchy. This patch sets the final queue path during initialization.
Queues must be stopped before reconfiguring, and queues can only be made from
leaf queues into parent queues and given children. Or, a totally new queue can
be added to an existing parent queue.
I did not include a unit test because this is a performance enhancement and did
not change the functionality. There are already many tests calling the
{{getQueuePath()}} and {{reinitialize}} APIs.
> Queue path is recomputed from scratch on every allocation
> -
>
> Key: YARN-6917
> URL: https://issues.apache.org/jira/browse/YARN-6917
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: capacityscheduler
>Affects Versions: 2.8.1
>Reporter: Jason Lowe
>Assignee: Eric Payne
>Priority: Minor
> Attachments: YARN-6917.001.patch
>
>
> As part of the discussion in YARN-6901 I noticed that we are recomputing a
> queue's path for every allocation. Currently getting the queue's path
> involves calling getQueuePath on the parent then building onto that string
> with the basename of the queue. In turn the parent's getQueuePath method
> does the same, so we end up spending time recomputing a string that will
> never change until a reconfiguration.
> Ideally the queue path should be computed once during queue initialization
> rather than on-demand.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-3254) HealthReport should include disk full information
[
https://issues.apache.org/jira/browse/YARN-3254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16117436#comment-16117436
]
Hadoop QA commented on YARN-3254:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
15s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
26s{color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
43s{color} | {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
in trunk has 5 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
14s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 16s{color} | {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager:
The patch generated 17 new + 35 unchanged - 0 fixed = 52 total (was 35)
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 13m
25s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 33m 2s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | YARN-3254 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12880716/YARN-3254-006.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux e8e075be5529 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12
13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / adb84f3 |
| Default Java | 1.8.0_131 |
| findbugs | v3.1.0-RC1 |
| findbugs |
https://builds.apache.org/job/PreCommit-YARN-Build/16754/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-warnings.html
|
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/16754/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/16754/testReport/ |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/16754/console |
| Powered by | Apache Yetus 0.6.0-SNAPSHOT
[jira] [Commented] (YARN-6668) Use cgroup to get container resource utilization
[
https://issues.apache.org/jira/browse/YARN-6668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16117430#comment-16117430
]
Hadoop QA commented on YARN-6668:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} YARN-1011 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
23s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
14s{color} | {color:green} YARN-1011 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
42s{color} | {color:green} YARN-1011 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
54s{color} | {color:green} YARN-1011 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
13s{color} | {color:green} YARN-1011 passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
10s{color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common in
YARN-1011 has 1 extant Findbugs warnings. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
52s{color} | {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
in YARN-1011 has 5 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
0s{color} | {color:green} YARN-1011 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
30s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 13m
22s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
34s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 70m 1s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | YARN-6668 |
| GITHUB PR | https://github.com/apache/hadoop/pull/241 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 991ffd8b10f1 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31
14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | YARN-1011 / 4c501b4 |
| Default Java | 1.8.0_131 |
| findbugs | v3.1.0-RC1 |
| findbugs |
https://builds.apache.org/job/PreCommit-YARN-Build/16751/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common-warnings.html
|
| findbugs |
[jira] [Commented] (YARN-6900) ZooKeeper based implementation of the FederationStateStore
[
https://issues.apache.org/jira/browse/YARN-6900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16117414#comment-16117414
]
Hadoop QA commented on YARN-6900:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
13s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
36s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
11s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
59s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
13s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
17s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
18s{color} | {color:red} hadoop-yarn-server-common in the patch failed. {color}
|
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 3m
1s{color} | {color:red} root in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 3m 1s{color}
| {color:red} root in the patch failed. {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 54s{color} | {color:orange} root: The patch generated 13 new + 246 unchanged
- 4 fixed = 259 total (was 250) {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
22s{color} | {color:red} hadoop-yarn-server-common in the patch failed. {color}
|
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
25s{color} | {color:red} hadoop-yarn-server-common in the patch failed. {color}
|
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
49s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
22s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
22s{color} | {color:red}
hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common
generated 4 new + 30 unchanged - 0 fixed = 34 total (was 30) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
25s{color} | {color:green}
hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager
generated 0 new + 348 unchanged - 4 fixed = 348 total (was 352) {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 7m 5s{color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 31s{color}
| {color:red} hadoop-yarn-api in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 24s{color}
| {color:red} hadoop-yarn-server-common in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 43m 25s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
24s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}139m 27s{color} |
{color:black} {color} |
\\
\\
||
[jira] [Updated] (YARN-6033) Add support for sections in container-executor configuration file
[ https://issues.apache.org/jira/browse/YARN-6033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-6033: - Attachment: YARN-6033.011.patch Attached ver.11 patch, cc: [~miklos.szeg...@cloudera.com] > Add support for sections in container-executor configuration file > - > > Key: YARN-6033 > URL: https://issues.apache.org/jira/browse/YARN-6033 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Reporter: Varun Vasudev >Assignee: Varun Vasudev > Attachments: YARN-6033.003.patch, YARN-6033.004.patch, > YARN-6033.005.patch, YARN-6033.006.patch, YARN-6033.007.patch, > YARN-6033.008.patch, YARN-6033.009.patch, YARN-6033.010.patch, > YARN-6033.011.patch, YARN-6033-YARN-5673.001.patch, > YARN-6033-YARN-5673.002.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6900) ZooKeeper based implementation of the FederationStateStore
[
https://issues.apache.org/jira/browse/YARN-6900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16117410#comment-16117410
]
Íñigo Goiri commented on YARN-6900:
---
Pushed the refactoring to HADOOP-14741.
Once that's done I'll rebase this patch.
Feel free to leave comments for the parts particular for this JIRA.
> ZooKeeper based implementation of the FederationStateStore
> --
>
> Key: YARN-6900
> URL: https://issues.apache.org/jira/browse/YARN-6900
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: federation, nodemanager, resourcemanager
>Reporter: Subru Krishnan
>Assignee: Íñigo Goiri
> Attachments: YARN-6900-002.patch, YARN-6900-003.patch,
> YARN-6900-YARN-2915-000.patch, YARN-6900-YARN-2915-001.patch
>
>
> YARN-5408 defines the unified {{FederationStateStore}} API. Currently we only
> support SQL based stores, this JIRA tracks adding a ZooKeeper based
> implementation for simplifying deployment as it's already popularly used for
> {{RMStateStore}}.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6930) Admins should be able to explicitly enable specific LinuxContainerRuntime in the NodeManager
[
https://issues.apache.org/jira/browse/YARN-6930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16117380#comment-16117380
]
Miklos Szegedi commented on YARN-6930:
--
Thank you for the patch, [~shaneku...@gmail.com]. Would you mind adding two
assertFalses verifying that Docker and Sandboxk are not allowed in .
{{testIsRuntimeAllowedDefault}}?
I am wondering whether it would be a good idea to specify the user as well not
just enabling a runtime in general. I could imagine that an admin allows Docker
runtime only for specific users first...
> Admins should be able to explicitly enable specific LinuxContainerRuntime in
> the NodeManager
>
>
> Key: YARN-6930
> URL: https://issues.apache.org/jira/browse/YARN-6930
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: nodemanager
>Reporter: Vinod Kumar Vavilapalli
>Assignee: Shane Kumpf
> Attachments: YARN-6930.001.patch
>
>
> Today, in the java land, all LinuxContainerRuntimes are always enabled when
> using LinuxContainerExecutor and the user can simply invoke anything that
> he/she wants - default, docker, java-sandbox.
> We should have a way for admins to explicitly enable only specific runtimes
> that he/she decides for the cluster. And by default, we should have
> everything other than the default one disabled.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5464) Server-Side NM Graceful Decommissioning with RM HA
[ https://issues.apache.org/jira/browse/YARN-5464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16117374#comment-16117374 ] Robert Kanter commented on YARN-5464: - I don't currently have the bandwidth. Go ahead [~djp]. Hopefully you'll find my wip patch to be helpful. > Server-Side NM Graceful Decommissioning with RM HA > -- > > Key: YARN-5464 > URL: https://issues.apache.org/jira/browse/YARN-5464 > Project: Hadoop YARN > Issue Type: Sub-task > Components: graceful >Reporter: Robert Kanter >Priority: Blocker > Attachments: YARN-5464.wip.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-3254) HealthReport should include disk full information
[ https://issues.apache.org/jira/browse/YARN-3254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suma Shivaprasad updated YARN-3254: --- Attachment: YARN-3254-006.patch Thanks [~sunilg]. Attached patched with review comments addressed. unused import java.util.NoSuchElementException --> Removed Java doc could also add param info as we ill new apis such as DirectoryCollection#getDirectoryErrorInfo etc Its better to use errorInformation instead of getValue. directoryErrorInfo.put(entry.getKey(), entry.getValue()); --> Fixed Could below message be kept at class level in LocalDirsHandlerService final String diskCapacityExceededErrorMsg --> Fixed One doubt in DirectoryCollection.checkDirs() As per latest patch, directoryErrorInfo is the super set which contains dirs from errorDirs, fullDirs and nothing more. Now while buildDiskErrorReport call, directoryCollection.isDiskUnHealthy is invoked for erroredLocal/LogDirsList and diskFullLocal/LogDirsList. I think this is not needed as contents from errorDirs and fillDirs are in that map for sure. In that case, could we skip this check and remove that api. Please correct me if I missed something here. -->Thanks for catching this. There was one case where the errorDirs was being updated but directoryErrorInfo was not being updated i.e in createNonExistentDirs which caused the lookup to return null. Have fixed that as part of the updated patch. Not sure if any other cases exist. Hence added a check to just be safe with the error information retrieval. > HealthReport should include disk full information > - > > Key: YARN-3254 > URL: https://issues.apache.org/jira/browse/YARN-3254 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager >Affects Versions: 2.6.0 >Reporter: Akira Ajisaka >Assignee: Suma Shivaprasad > Fix For: 3.0.0-beta1 > > Attachments: Screen Shot 2015-02-24 at 17.57.39.png, Screen Shot > 2015-02-25 at 14.38.10.png, YARN-3254-001.patch, YARN-3254-002.patch, > YARN-3254-003.patch, YARN-3254-004.patch, YARN-3254-005.patch, > YARN-3254-006.patch > > > When a NodeManager's local disk gets almost full, the NodeManager sends a > health report to ResourceManager that "local/log dir is bad" and the message > is displayed on ResourceManager Web UI. It's difficult for users to detect > why the dir is bad. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-6920) Fix TestNMClient failure due to YARN-6706
[
https://issues.apache.org/jira/browse/YARN-6920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16117367#comment-16117367
]
Arun Suresh edited comment on YARN-6920 at 8/7/17 9:48 PM:
---
Kicked Jenkins off again for this patch - to verify that the last successful
run was not just a one off thing.
Will commit it after, if successful - based on [~jianhe]'s lgtm
was (Author: asuresh):
Kicked Jenkins off again for this patch - to verify that the last successful
was not just a one off thing.
Will commit it after, if successful - based on [~jianhe]'s lgtm
> Fix TestNMClient failure due to YARN-6706
> -
>
> Key: YARN-6920
> URL: https://issues.apache.org/jira/browse/YARN-6920
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Arun Suresh
>Assignee: Arun Suresh
> Attachments: YARN-6920.001.patch, YARN-6920.002.patch,
> YARN-6920.003.patch, YARN-6920.004.patch
>
>
> Looks like {{TestNMClient}} has been failing for a while. Opening this JIRA
> to track the fix.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6920) Fix TestNMClient failure due to YARN-6706
[
https://issues.apache.org/jira/browse/YARN-6920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16117367#comment-16117367
]
Arun Suresh commented on YARN-6920:
---
Kicked Jenkins off again for this patch - to verify that the last successful
was not just a one off thing.
Will commit it after, if successful - based on [~jianhe]'s lgtm
> Fix TestNMClient failure due to YARN-6706
> -
>
> Key: YARN-6920
> URL: https://issues.apache.org/jira/browse/YARN-6920
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Arun Suresh
>Assignee: Arun Suresh
> Attachments: YARN-6920.001.patch, YARN-6920.002.patch,
> YARN-6920.003.patch, YARN-6920.004.patch
>
>
> Looks like {{TestNMClient}} has been failing for a while. Opening this JIRA
> to track the fix.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6930) Admins should be able to explicitly enable specific LinuxContainerRuntime in the NodeManager
[ https://issues.apache.org/jira/browse/YARN-6930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shane Kumpf updated YARN-6930: -- Attachment: YARN-6930.001.patch > Admins should be able to explicitly enable specific LinuxContainerRuntime in > the NodeManager > > > Key: YARN-6930 > URL: https://issues.apache.org/jira/browse/YARN-6930 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager >Reporter: Vinod Kumar Vavilapalli >Assignee: Shane Kumpf > Attachments: YARN-6930.001.patch > > > Today, in the java land, all LinuxContainerRuntimes are always enabled when > using LinuxContainerExecutor and the user can simply invoke anything that > he/she wants - default, docker, java-sandbox. > We should have a way for admins to explicitly enable only specific runtimes > that he/she decides for the cluster. And by default, we should have > everything other than the default one disabled. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6033) Add support for sections in container-executor configuration file
[ https://issues.apache.org/jira/browse/YARN-6033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16117351#comment-16117351 ] Wangda Tan commented on YARN-6033: -- Thanks [~miklos.szeg...@cloudera.com], will update patch shortly. > Add support for sections in container-executor configuration file > - > > Key: YARN-6033 > URL: https://issues.apache.org/jira/browse/YARN-6033 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Reporter: Varun Vasudev >Assignee: Varun Vasudev > Attachments: YARN-6033.003.patch, YARN-6033.004.patch, > YARN-6033.005.patch, YARN-6033.006.patch, YARN-6033.007.patch, > YARN-6033.008.patch, YARN-6033.009.patch, YARN-6033.010.patch, > YARN-6033-YARN-5673.001.patch, YARN-6033-YARN-5673.002.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6920) Fix TestNMClient failure due to YARN-6706
[
https://issues.apache.org/jira/browse/YARN-6920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16117336#comment-16117336
]
Haibo Chen commented on YARN-6920:
--
[~asuresh] Seems like I am hitting another issue (due to race condition) before
the one you pointed out. Your change looks good to me. But since I cannot
download and verify, I'll let [~jianhe] +1 and commit.
> Fix TestNMClient failure due to YARN-6706
> -
>
> Key: YARN-6920
> URL: https://issues.apache.org/jira/browse/YARN-6920
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Arun Suresh
>Assignee: Arun Suresh
> Attachments: YARN-6920.001.patch, YARN-6920.002.patch,
> YARN-6920.003.patch, YARN-6920.004.patch
>
>
> Looks like {{TestNMClient}} has been failing for a while. Opening this JIRA
> to track the fix.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6920) Fix TestNMClient failure due to YARN-6706
[
https://issues.apache.org/jira/browse/YARN-6920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16117334#comment-16117334
]
Jian He commented on YARN-6920:
---
yep, the patch lgtm
> Fix TestNMClient failure due to YARN-6706
> -
>
> Key: YARN-6920
> URL: https://issues.apache.org/jira/browse/YARN-6920
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Arun Suresh
>Assignee: Arun Suresh
> Attachments: YARN-6920.001.patch, YARN-6920.002.patch,
> YARN-6920.003.patch, YARN-6920.004.patch
>
>
> Looks like {{TestNMClient}} has been failing for a while. Opening this JIRA
> to track the fix.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6920) Fix TestNMClient failure due to YARN-6706
[
https://issues.apache.org/jira/browse/YARN-6920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16117322#comment-16117322
]
Arun Suresh commented on YARN-6920:
---
Sure.. Raised YARN-6963 to track that.
Let me know if this patch is fine and I'll check it in.
> Fix TestNMClient failure due to YARN-6706
> -
>
> Key: YARN-6920
> URL: https://issues.apache.org/jira/browse/YARN-6920
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Arun Suresh
>Assignee: Arun Suresh
> Attachments: YARN-6920.001.patch, YARN-6920.002.patch,
> YARN-6920.003.patch, YARN-6920.004.patch
>
>
> Looks like {{TestNMClient}} has been failing for a while. Opening this JIRA
> to track the fix.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6963) Prevent other containers from staring when a container is re-initializing
[ https://issues.apache.org/jira/browse/YARN-6963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun Suresh updated YARN-6963: -- Issue Type: Improvement (was: Bug) > Prevent other containers from staring when a container is re-initializing > - > > Key: YARN-6963 > URL: https://issues.apache.org/jira/browse/YARN-6963 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Arun Suresh >Assignee: Arun Suresh > > Further to discussions in YARN-6920. > Container re-initialization will lead to momentary relinquishing of NM > resources when the container is brought down followed by re-claiming of the > same resources when it is re-launched. If there are Opportunistic containers > in the queue, it can lead to un-necessary churn if one of those opportunistic > containers are started and immediately killed. > This JIRA tracks changes required to prevent the above by ensuring the > resources for a container are 'locked' for the during of the container > lifetime - including the time it takes for a re-initialization. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-6963) Prevent other containers from staring when a container is re-initializing
Arun Suresh created YARN-6963: - Summary: Prevent other containers from staring when a container is re-initializing Key: YARN-6963 URL: https://issues.apache.org/jira/browse/YARN-6963 Project: Hadoop YARN Issue Type: Bug Reporter: Arun Suresh Assignee: Arun Suresh Further to discussions in YARN-6920. Container re-initialization will lead to momentary relinquishing of NM resources when the container is brought down followed by re-claiming of the same resources when it is re-launched. If there are Opportunistic containers in the queue, it can lead to un-necessary churn if one of those opportunistic containers are started and immediately killed. This JIRA tracks changes required to prevent the above by ensuring the resources for a container are 'locked' for the during of the container lifetime - including the time it takes for a re-initialization. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4161) Capacity Scheduler : Assign single or multiple containers per heart beat driven by configuration
[ https://issues.apache.org/jira/browse/YARN-4161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16117289#comment-16117289 ] Wangda Tan commented on YARN-4161: -- Test failure is not related, pushing this to branch-2. > Capacity Scheduler : Assign single or multiple containers per heart beat > driven by configuration > > > Key: YARN-4161 > URL: https://issues.apache.org/jira/browse/YARN-4161 > Project: Hadoop YARN > Issue Type: New Feature > Components: capacity scheduler >Reporter: Mayank Bansal >Assignee: Wei Yan > Labels: oct16-medium > Fix For: 3.0.0-beta1 > > Attachments: YARN-4161.002.patch, YARN-4161.003.patch, > YARN-4161.004.patch, YARN-4161.005.patch, YARN-4161.006.patch, > YARN-4161.branch-2.006.patch, YARN-4161.patch, YARN-4161.patch.1 > > > Capacity Scheduler right now schedules multiple containers per heart beat if > there are more resources available in the node. > This approach works fine however in some cases its not distribute the load > across the cluster hence throughput of the cluster suffers. I am adding > feature to drive that using configuration by that we can control the number > of containers assigned per heart beat. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6240) TestCapacityScheduler.testRefreshQueuesWithQueueDelete fails randomly
[
https://issues.apache.org/jira/browse/YARN-6240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16117288#comment-16117288
]
Wangda Tan commented on YARN-6240:
--
+1 to push this patch to code base. I saw this UT failure occurred many times
in Jenkins result.
> TestCapacityScheduler.testRefreshQueuesWithQueueDelete fails randomly
> -
>
> Key: YARN-6240
> URL: https://issues.apache.org/jira/browse/YARN-6240
> Project: Hadoop YARN
> Issue Type: Test
> Components: test
>Reporter: Sunil G
>Assignee: Naganarasimha G R
> Attachments: YARN-6240.001.patch
>
>
> *Error Message*
> Expected to NOT throw exception when refresh queue tries to delete a queue
> WITHOUT running apps
> Link
> [here|https://builds.apache.org/job/PreCommit-YARN-Build/15092/testReport/org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity/TestCapacityScheduler/testRefreshQueuesWithQueueDelete/]
> *Stacktrace*
> {code}
> java.lang.AssertionError: Expected to NOT throw exception when refresh queue
> tries to delete a queue WITHOUT running apps
> at org.junit.Assert.fail(Assert.java:88)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler.testRefreshQueuesWithQueueDelete(TestCapacityScheduler.java:3875)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4161) Capacity Scheduler : Assign single or multiple containers per heart beat driven by configuration
[
https://issues.apache.org/jira/browse/YARN-4161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16117270#comment-16117270
]
Hadoop QA commented on YARN-4161:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 17m
53s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} branch-2 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
5s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
39s{color} | {color:green} branch-2 passed with JDK v1.8.0_144 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
35s{color} | {color:green} branch-2 passed with JDK v1.7.0_131 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
27s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
43s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
19s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
22s{color} | {color:green} branch-2 passed with JDK v1.8.0_144 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
26s{color} | {color:green} branch-2 passed with JDK v1.7.0_131 {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
34s{color} | {color:green} the patch passed with JDK v1.8.0_144 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
34s{color} | {color:green} the patch passed with JDK v1.7.0_131 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
20s{color} | {color:green} the patch passed with JDK v1.8.0_144 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
22s{color} | {color:green} the patch passed with JDK v1.7.0_131 {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 50m 18s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed with JDK
v1.7.0_131. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}134m 36s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.7.0_131 Failed junit tests |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler |
| | hadoop.yarn.server.resourcemanager.TestLeaderElectorService |
| | hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:5e40efe |
| JIRA Issue | YARN-4161 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12880681/YARN-4161.branch-2.006.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 577cc68bf93b 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31
14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
|
[jira] [Commented] (YARN-6668) Use cgroup to get container resource utilization
[ https://issues.apache.org/jira/browse/YARN-6668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16117264#comment-16117264 ] Haibo Chen commented on YARN-6668: -- Thanks [~miklos.szeg...@cloudera.com] for the update! I have a few follow-up knits: 1) The change in ResourceCalculatorProcessTree.java is no longer needed. 2) Can we add @Override annotation to CGroupsHandlerImpl.getControllerPath()? 3) The CGroupsResoureCalculator now takes precedence over user-configuration. 4) There's an unused import java.io.FileNotFoundException; in CGroupsResourceCalculator 5) ContainersMonitorImpl is missing an import of YarnException, which leads to compilation failure 6) performance is misspelled as "perforance" in ContainersMonitorImpl.getResourceCalculatorProcessTree() 7) Now that we separate getCgroupHandlers from initCgroupHandlers in ResourceHandlerModule, and calls getCgroupHandlers() in CGroupsResourceCalculator to determine whether it is available. I think we have an implicit assumption here that CGroup is initialized before CGroupsResourceCalculator.isAvailable() is called. Can we add a comment in isAvailable to make it more obvious? 8) In CGroupsResourceCalculator constructor, should we initialize jiffyLengthMs first and then cpuTimeTracker to avoid the cases where cpuTimeTracker get -1 as the jiffyLength whereas jiffyLengthMs is set to 10? 9) We call setCGroupFilePaths() in constructor which reads file. Can we move that out into an init() method instead? 10) the lock around firstError is unnecessary if we make firstError volatile. 11) TestCGroupsResourceCalculator.testNoPid() is misnamed in that the constructor actually takes a pid '1234'. Likewise, testNoCGgroup is testing calculator.getRssMemorySize(0) returns UNAVAILABLE, right? 12) How about we reference '/tmp/' + this.getClass().getName() as a static field profcBase? Optionally, we could then move all deletions of procfs into a teardown method. 13) testNoMemoryCGgroupMount can use a ExpectedException to replace the AssertTrue in the catch block. > Use cgroup to get container resource utilization > > > Key: YARN-6668 > URL: https://issues.apache.org/jira/browse/YARN-6668 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Affects Versions: 3.0.0-alpha3 >Reporter: Haibo Chen >Assignee: Miklos Szegedi > Attachments: YARN-6668.000.patch, YARN-6668.001.patch, > YARN-6668.002.patch > > > Container Monitor relies on proc file system to get container resource > utilization, which is not as efficient as reading cgroup accounting. We > should in NM, when cgroup is enabled, read cgroup stats instead. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6920) Fix TestNMClient failure due to YARN-6706
[
https://issues.apache.org/jira/browse/YARN-6920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16117260#comment-16117260
]
Jian He commented on YARN-6920:
---
[~asuresh], yep, I'm fine to fix it in a separate jira.
I would think the optimization is required in addition to YARN-5972, because
the container upgrade process is expected to be quick (we have been working
towards this goal to make localization happen before upgrade). And pause and
unpause may also introduce additional delay which can be avoided.
> Fix TestNMClient failure due to YARN-6706
> -
>
> Key: YARN-6920
> URL: https://issues.apache.org/jira/browse/YARN-6920
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Arun Suresh
>Assignee: Arun Suresh
> Attachments: YARN-6920.001.patch, YARN-6920.002.patch,
> YARN-6920.003.patch, YARN-6920.004.patch
>
>
> Looks like {{TestNMClient}} has been failing for a while. Opening this JIRA
> to track the fix.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6033) Add support for sections in container-executor configuration file
[
https://issues.apache.org/jira/browse/YARN-6033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16117247#comment-16117247
]
Miklos Szegedi commented on YARN-6033:
--
[~wangda], sorry there is an issue in the latest change:
{code}
70 cleanup:
71if (failed) {
72 free_values(return_values);
73}
74
75return return_values;
76 }
{code}
We return a freed pointer if failed is 1 instead of NULL.
Another one: read_section_entry: buffer is never freed I think
> Add support for sections in container-executor configuration file
> -
>
> Key: YARN-6033
> URL: https://issues.apache.org/jira/browse/YARN-6033
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager
>Reporter: Varun Vasudev
>Assignee: Varun Vasudev
> Attachments: YARN-6033.003.patch, YARN-6033.004.patch,
> YARN-6033.005.patch, YARN-6033.006.patch, YARN-6033.007.patch,
> YARN-6033.008.patch, YARN-6033.009.patch, YARN-6033.010.patch,
> YARN-6033-YARN-5673.001.patch, YARN-6033-YARN-5673.002.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6900) ZooKeeper based implementation of the FederationStateStore
[
https://issues.apache.org/jira/browse/YARN-6900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16117220#comment-16117220
]
Subru Krishnan commented on YARN-6900:
--
[~elgoiri], thanks for refactoring, it looks good. Since it's non-trivial and
is consolidation at Hadoop level, I have opened HADOOP-14741 to track the
refactoring. So can you kindly move it there as we'll use this just for
ZK-based {{FederationStateStore}} impl.
> ZooKeeper based implementation of the FederationStateStore
> --
>
> Key: YARN-6900
> URL: https://issues.apache.org/jira/browse/YARN-6900
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: federation, nodemanager, resourcemanager
>Reporter: Subru Krishnan
>Assignee: Inigo Goiri
> Attachments: YARN-6900-002.patch, YARN-6900-003.patch,
> YARN-6900-YARN-2915-000.patch, YARN-6900-YARN-2915-001.patch
>
>
> YARN-5408 defines the unified {{FederationStateStore}} API. Currently we only
> support SQL based stores, this JIRA tracks adding a ZooKeeper based
> implementation for simplifying deployment as it's already popularly used for
> {{RMStateStore}}.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6920) Fix TestNMClient failure due to YARN-6706
[
https://issues.apache.org/jira/browse/YARN-6920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16117208#comment-16117208
]
Arun Suresh commented on YARN-6920:
---
Ping [~haibo.chen] / [~jianhe]. Do you guys want me to make the [afore
mentioned|https://issues.apache.org/jira/browse/YARN-6920?focusedCommentId=16114021=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16114021]
optimization here - or should I raise a JIRA to handle this later. I would
prefer the later though, if atleast to not hold-up getting {{TestNMClient}} to
pass again in trunk.
> Fix TestNMClient failure due to YARN-6706
> -
>
> Key: YARN-6920
> URL: https://issues.apache.org/jira/browse/YARN-6920
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Arun Suresh
>Assignee: Arun Suresh
> Attachments: YARN-6920.001.patch, YARN-6920.002.patch,
> YARN-6920.003.patch, YARN-6920.004.patch
>
>
> Looks like {{TestNMClient}} has been failing for a while. Opening this JIRA
> to track the fix.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6897) Refactoring RMWebServices by moving some util methods to RMWebAppUtil
[
https://issues.apache.org/jira/browse/YARN-6897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16117188#comment-16117188
]
Hadoop QA commented on YARN-6897:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
14s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} branch-2 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
20s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
26s{color} | {color:green} branch-2 passed with JDK v1.8.0_144 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
33s{color} | {color:green} branch-2 passed with JDK v1.7.0_131 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
21s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
35s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
5s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
17s{color} | {color:green} branch-2 passed with JDK v1.8.0_144 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
22s{color} | {color:green} branch-2 passed with JDK v1.7.0_131 {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
24s{color} | {color:green} the patch passed with JDK v1.8.0_144 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
29s{color} | {color:green} the patch passed with JDK v1.7.0_131 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
19s{color} | {color:green}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 0 new + 12 unchanged - 1 fixed = 12 total (was 13) {color}
|
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
17s{color} | {color:green}
hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_144
with JDK v1.8.0_144 generated 0 new + 858 unchanged - 3 fixed = 858 total (was
861) {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
21s{color} | {color:red}
hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_131
with JDK v1.7.0_131 generated 1 new + 0 unchanged - 1 fixed = 1 total (was 1)
{color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 47m 6s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed with JDK
v1.7.0_131. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}106m 56s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.7.0_131 Failed junit tests |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler |
| | hadoop.yarn.server.resourcemanager.TestRMEmbeddedElector |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:5e40efe |
| JIRA
[jira] [Commented] (YARN-6852) [YARN-6223] Native code changes to support isolate GPU devices by using CGroups
[
https://issues.apache.org/jira/browse/YARN-6852?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16117175#comment-16117175
]
Miklos Szegedi commented on YARN-6852:
--
Thank you [~wangda],
I have a second batch.
I could not find a doc about this, so I ask. Is there a reason to have a
disable list of devices, instead of an enable list?
command-line-parser.c, etc: I think it is more common to have a_b.c style file
naming. For example in the Linux kernel: attribute_container.c
{code}
int handle_gpu_request(update_cgroups_parameters_func func,
143const char* module_name, int argc, char** argv) {
{code}
Since they are not argv anymore I would use the names paramc, params instead of
argc, arg
{code}
int* minor_devices;
{code}
I would call internal_handle_gpu_request, only if minor_devices is initialized.
Currently it may contain any garbage on the stack, if ‘e’ never hits.
{code}
191 default:
192fprintf(ERRORFILE,
193 "Unknown option in gpu command character %d %c, optionindex =
%d\n",
194 c, c, optind);
195return -1;
196break;
{code}
No need to break;
{code}
if (0 == strlen(container_id)) {
{code}
container_id[0]==0
{code}
strncpy(container_id, optarg, MAX_CONTAINER_ID_LEN)
{code}
strncpy does not null terminate the string if the length equals to MAX_CON...
{code}
free(full_path);
{code}
It is a common practice to
{code}
… goto cleanup;
cleanup:
free(full_path);
return X;
{code}
-
{code}
if (!major_number_str || 0 == strlen(major_number_str)) {
{code}
This could be major_number_str[0]==0
{code}
61// Default major number of Nvidia devices
62major_device_number = 195;
{code}
It might be nicer to have a #define for this number
{code}
72 if (!allowed_minor_numbers_str || strlen(allowed_minor_numbers_str)) {
73allowed_minor_numbers = NULL;
{code}
Bug: strlen==0 (!) or allowed_minor_numbers_str[0]==0
{code}
get_section_value return values are not freed in internal_handle_gpu_request
{code}
200 if (0 == strlen(container_id)) {
{code}
This could be container_id[0]==0
{code}
164 int* minor_devices;
{code}
I think this is never freed.
parse_commandline_opts may leak opts, opts->keys and opts-> values when
returning -1 on errors.
{code}
41 if (!opts->keys || !opts->values) {
42fprintf(ERRORFILE, "Failed to malloc keys or values of opts\n");
43return NULL;
44 }
{code}
You still need to free keys if values is NULL
{code}
51 if (!(known_parameters && required && has_values)) {
{code}
Ideally, this is a check at the beginning of the function.
{code}
64// make sure param_name start with "--"
65if (0 != strncmp("--", param_name, 2)) {
66 fprintf(ERRORFILE, "option %s is not started with \"--\"\n",
param_name);
67 return NULL;
68}
82if (param_idx < 0) {
83 fprintf(ERRORFILE, "cannot find parameter %s from known
parameters\n", param_name);
84 return NULL;
85}
{code}
param_name is leaked.
Is parse_commandline_opts used anywhere? If not, please remove it from this
jira.
{code}
24/*
25 * if all chars in the input str are numbers
26 * return true/false
27 */
28static int all_numbers(char* input) {
29 int len = strlen(input);
30
31 for (int i = 0; i < len; i++) {
32if (input[i] < '0' || input[i] > '9') {
33 return 0;
34}
35 }
36 return 1;
37}
{code}
This should be like the code below, otherwise we pass through the string twice:
{code}
24/*
25 * if all chars in the input str are numbers
26 * return true/false
27 */
28static int all_numbers(char* input) {
31 for (; input[0] != 0; input++) {
32if (input[0] < '0' || input[0] > '9') {
33 return 0;
34}
35 }
36 return 1;
37}
{code}
{code}
39int get_numbers_split_by_comma(char* input, int** numbers, size_t*
ret_n_numbers) {
40 size_t n_numbers = 1;
41 for (int i = 0; i < strlen(input); i++) {
42if (input[i] == ',') {
43 n_numbers++;
44}
45 }
46
47 (*numbers) = malloc(sizeof(int) * n_numbers);
48 if (!(*numbers)) {
49return -1;
50 }
51
52 char* input_cpy = malloc(strlen(input));
53 strcpy(input_cpy, input);
54
55 char* p = strtok(input_cpy, ",");
56 int idx = 0;
57 while (p != NULL) {
58int n = atoi(p);
59(*numbers)[idx] = n;
60p = strtok(NULL, ",");
61idx++;
62 }
63
64 free(input_cpy);
65 *ret_n_numbers = n_numbers;
66
67 return 0;
68}
{code}
get_numbers_split_by_comma: Please do not do strlen at every character. It will
be an O(n^2) algorithm from an O(n).
get_numbers_split_by_comma: You may actually return an array with garbage at
the end, since a double comma (,,) will not trigger a new
[jira] [Commented] (YARN-65) Reduce RM app memory footprint once app has completed
[
https://issues.apache.org/jira/browse/YARN-65?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16117174#comment-16117174
]
Hadoop QA commented on YARN-65:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
22s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
4s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
23s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 25s{color} | {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 3 new + 137 unchanged - 1 fixed = 140 total (was 138)
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 2 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 46m 37s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 70m 50s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.resourcemanager.scheduler.fair.TestFSAppStarvation |
| Timed out junit tests |
org.apache.hadoop.yarn.server.resourcemanager.TestSubmitApplicationWithRMHA |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | YARN-65 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12880680/YARN-65.005.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 5304b510a07b 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31
14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / adb84f3 |
| Default Java | 1.8.0_131 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/16749/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
|
| whitespace |
https://builds.apache.org/job/PreCommit-YARN-Build/16749/artifact/patchprocess/whitespace-eol.txt
|
| unit |
https://builds.apache.org/job/PreCommit-YARN-Build/16749/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/16749/testReport/ |
| modules | C:
[jira] [Updated] (YARN-6900) ZooKeeper based implementation of the FederationStateStore
[
https://issues.apache.org/jira/browse/YARN-6900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Inigo Goiri updated YARN-6900:
--
Attachment: YARN-6900-003.patch
Created {{ZKManager}}. Let me know what are the thought on the refactoring.
> ZooKeeper based implementation of the FederationStateStore
> --
>
> Key: YARN-6900
> URL: https://issues.apache.org/jira/browse/YARN-6900
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: federation, nodemanager, resourcemanager
>Reporter: Subru Krishnan
>Assignee: Inigo Goiri
> Attachments: YARN-6900-002.patch, YARN-6900-003.patch,
> YARN-6900-YARN-2915-000.patch, YARN-6900-YARN-2915-001.patch
>
>
> YARN-5408 defines the unified {{FederationStateStore}} API. Currently we only
> support SQL based stores, this JIRA tracks adding a ZooKeeper based
> implementation for simplifying deployment as it's already popularly used for
> {{RMStateStore}}.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-6879) TestLeafQueue.testDRFUserLimits() has commented out code
[ https://issues.apache.org/jira/browse/YARN-6879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Templeton reassigned YARN-6879: -- Assignee: Angela Wang (was: Daniel Templeton) > TestLeafQueue.testDRFUserLimits() has commented out code > > > Key: YARN-6879 > URL: https://issues.apache.org/jira/browse/YARN-6879 > Project: Hadoop YARN > Issue Type: Improvement > Components: capacity scheduler, test >Affects Versions: 3.0.0-alpha4 >Reporter: Daniel Templeton >Assignee: Angela Wang >Priority: Trivial > Labels: newbie > > The commented-out code should be deleted. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6875) New aggregated log file format for YARN log aggregation.
[ https://issues.apache.org/jira/browse/YARN-6875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16117123#comment-16117123 ] Jason Lowe commented on YARN-6875: -- I'm not clear on how the local index file would work in practice. One use case for aggregated logs is to run a cluster job that analyzes the log files in HDFS. How would a cluster job handle accessing the local index file for one of the log aggregation files? Is it proxying to the local index file via the nodemanager corresponding to the log aggregation file? If so then that means the job now needs sufficient credentials to be able to authenticate to an NM to read log aggregation files. If the NM can fixup failed appends then the main difference between the new proposal and one that simply scans when it can't immediately find the metainfo is the likelihood the reader will read during an append. The NM crashes/disappears failure case is the same, so if we agree that trying to read during a massive append is going to be similarly rare then I'd rather avoid having log readers contacting the NM if possible. > New aggregated log file format for YARN log aggregation. > > > Key: YARN-6875 > URL: https://issues.apache.org/jira/browse/YARN-6875 > Project: Hadoop YARN > Issue Type: New Feature >Reporter: Xuan Gong >Assignee: Xuan Gong > Attachments: YARN-6875-NewLogAggregationFormat-design-doc.pdf > > > T-file is the underlying log format for the aggregated logs in YARN. We have > seen several performance issues, especially for very large log files. > We will introduce a new log format which have better performance for large > log files. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6033) Add support for sections in container-executor configuration file
[
https://issues.apache.org/jira/browse/YARN-6033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16117084#comment-16117084
]
Hadoop QA commented on YARN-6033:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
15s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 7 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
18s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 0m
39s{color} | {color:green}
hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager
generated 0 new + 0 unchanged - 1 fixed = 0 total (was 1) {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 13m
23s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 32m 7s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | YARN-6033 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12880672/YARN-6033.010.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit xml cc |
| uname | Linux 533625d26fdb 3.13.0-119-generic #166-Ubuntu SMP Wed May 3
12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / adb84f3 |
| Default Java | 1.8.0_131 |
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/16747/testReport/ |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/16747/console |
| Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Add support for sections in container-executor configuration file
> -
>
> Key: YARN-6033
> URL: https://issues.apache.org/jira/browse/YARN-6033
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager
>Reporter: Varun Vasudev
>Assignee: Varun Vasudev
> Attachments: YARN-6033.003.patch, YARN-6033.004.patch,
> YARN-6033.005.patch, YARN-6033.006.patch, YARN-6033.007.patch,
> YARN-6033.008.patch, YARN-6033.009.patch, YARN-6033.010.patch,
> YARN-6033-YARN-5673.001.patch, YARN-6033-YARN-5673.002.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
