[jira] [Commented] (YARN-7013) merge related work for YARN-3926 branch
[
https://issues.apache.org/jira/browse/YARN-7013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16148482#comment-16148482
]
Hadoop QA commented on YARN-7013:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
22s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 53 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
29s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 16m
10s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 8m
33s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests
{color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
47s{color} | {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
in trunk has 1 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 5m
38s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
18s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 11m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 11m
24s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
2m 40s{color} | {color:orange} root: The patch generated 71 new + 2355
unchanged - 52 fixed = 2426 total (was 2407) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 8m
58s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
18s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests
{color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
18s{color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
46s{color} | {color:green} hadoop-yarn-project_hadoop-yarn generated 0 new +
5500 unchanged - 4 fixed = 5500 total (was 5504) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
24s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
42s{color} | {color:green} hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common
generated 0 new + 4510 unchanged - 4 fixed = 4510 total (was 4514) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
23s{color} | {color:green} hadoop-yarn-server-common in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
23s{color} | {color:green}
[jira] [Updated] (YARN-7127) Run yarn-native-services branch against trunk
[ https://issues.apache.org/jira/browse/YARN-7127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jian He updated YARN-7127: -- Attachment: YARN-7127.02.patch > Run yarn-native-services branch against trunk > - > > Key: YARN-7127 > URL: https://issues.apache.org/jira/browse/YARN-7127 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jian He >Assignee: Jian He > Attachments: YARN-7127.01.patch, YARN-7127.02.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6721) container-executor should have stack checking
[
https://issues.apache.org/jira/browse/YARN-6721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16148480#comment-16148480
]
Hadoop QA commented on YARN-6721:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 19m
5s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
28s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 14m
21s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 50m 39s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:71bbb86 |
| JIRA Issue | YARN-6721 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12884515/YARN-6721.00.patch |
| Optional Tests | asflicense compile cc mvnsite javac unit |
| uname | Linux 9434ec6b846c 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31
14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 71bbb86 |
| Default Java | 1.8.0_144 |
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/17225/testReport/ |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/17225/console |
| Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> container-executor should have stack checking
> -
>
> Key: YARN-6721
> URL: https://issues.apache.org/jira/browse/YARN-6721
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: nodemanager, security
>Reporter: Allen Wittenauer
>Assignee: Allen Wittenauer
>Priority: Critical
> Labels: security
> Attachments: YARN-6721.00.patch
>
>
> As per https://www.qualys.com/2017/06/19/stack-clash/stack-clash.txt and
> given that container-executor is setuid, it should be compiled with stack
> checking if the compiler supports such features. (-fstack-check on gcc,
> -fsanitize=safe-stack on clang, -xcheck=stkovf on "Oracle Solaris Studio",
> others as we find them, ...)
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7013) merge related work for YARN-3926 branch
[
https://issues.apache.org/jira/browse/YARN-7013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16148471#comment-16148471
]
Hadoop QA commented on YARN-7013:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 53 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
41s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
59s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 8m
38s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests
{color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
55s{color} | {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
in trunk has 1 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 5m
37s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
20s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 11m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 11m
20s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
2m 39s{color} | {color:orange} root: The patch generated 66 new + 2350
unchanged - 52 fixed = 2416 total (was 2402) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 8m
28s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
21s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests
{color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
32s{color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 1m
52s{color} | {color:red} hadoop-yarn-project_hadoop-yarn generated 1 new + 5500
unchanged - 4 fixed = 5501 total (was 5504) {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
24s{color} | {color:red} hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api
generated 1 new + 123 unchanged - 0 fixed = 124 total (was 123) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
42s{color} | {color:green} hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common
generated 0 new + 4510 unchanged - 4 fixed = 4510 total (was 4514) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
23s{color} | {color:green} hadoop-yarn-server-common in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color}
[jira] [Commented] (YARN-7139) FairScheduler: finished applications are always restored to default queue
[
https://issues.apache.org/jira/browse/YARN-7139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16148467#comment-16148467
]
Hadoop QA commented on YARN-7139:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
20s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
16s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
3s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
22s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
26s{color} | {color:green}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 0 new + 134 unchanged - 1 fixed = 134 total (was 135)
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 44m 38s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 67m 24s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairSchedulerPreemption |
| |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation |
| | hadoop.yarn.server.resourcemanager.scheduler.fair.TestFSLeafQueue |
| |
hadoop.yarn.server.resourcemanager.reservation.TestFairSchedulerPlanFollower |
| |
hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairSchedulerFairShare |
| | hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairScheduler |
| | hadoop.yarn.server.resourcemanager.scheduler.fair.TestAppRunnability |
| | hadoop.yarn.server.resourcemanager.scheduler.fair.TestFSAppStarvation |
| | hadoop.yarn.server.resourcemanager.scheduler.fair.TestFSAppAttempt |
| |
hadoop.yarn.server.resourcemanager.scheduler.fair.TestContinuousScheduling |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | YARN-7139 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12884585/YARN-7139.01.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux d1ff298ef932 3.13.0-119-generic #166-Ubuntu SMP Wed May 3
12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / ce79f7b |
| Default Java | 1.8.0_144 |
| findbugs | v3.1.0-RC1 |
| unit |
[jira] [Updated] (YARN-7139) FairScheduler: finished applications are always restored to default queue
[ https://issues.apache.org/jira/browse/YARN-7139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wilfred Spiegelenburg updated YARN-7139: Attachment: YARN-7139.01.patch > FairScheduler: finished applications are always restored to default queue > - > > Key: YARN-7139 > URL: https://issues.apache.org/jira/browse/YARN-7139 > Project: Hadoop YARN > Issue Type: Bug > Components: fairscheduler >Affects Versions: 2.8.1 >Reporter: Wilfred Spiegelenburg >Assignee: Wilfred Spiegelenburg > Attachments: YARN-7139.01.patch > > > The queue an application gets submitted to is defined by the placement policy > in the FS. The placement policy returns the queue and the application object > is updated. When an application is stored in the state store the application > submission context is used which has not been updated after the placement > rules have run. > This means that the original queue from the submission is still stored which > is the incorrect queue. On restore we then read back the wrong queue and > display the wrong queue in the RM web UI. > We should update the submission context after we have run the placement > policies to make sure that we store the correct queue for the application. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-7139) FairScheduler: finished applications are always restored to default queue
Wilfred Spiegelenburg created YARN-7139: --- Summary: FairScheduler: finished applications are always restored to default queue Key: YARN-7139 URL: https://issues.apache.org/jira/browse/YARN-7139 Project: Hadoop YARN Issue Type: Bug Components: fairscheduler Affects Versions: 2.8.1 Reporter: Wilfred Spiegelenburg Assignee: Wilfred Spiegelenburg The queue an application gets submitted to is defined by the placement policy in the FS. The placement policy returns the queue and the application object is updated. When an application is stored in the state store the application submission context is used which has not been updated after the placement rules have run. This means that the original queue from the submission is still stored which is the incorrect queue. On restore we then read back the wrong queue and display the wrong queue in the RM web UI. We should update the submission context after we have run the placement policies to make sure that we store the correct queue for the application. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7132) FairScheduler.initScheduler() contains a surprising unary plus
[
https://issues.apache.org/jira/browse/YARN-7132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16148389#comment-16148389
]
Hadoop QA commented on YARN-7132:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
21s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
1s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 43m 23s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 65m 57s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | YARN-7132 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12884579/YARN-7132.001.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 772311f8bd8b 3.13.0-119-generic #166-Ubuntu SMP Wed May 3
12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / ce79f7b |
| Default Java | 1.8.0_144 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-YARN-Build/17223/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/17223/testReport/ |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/17223/console |
| Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> FairScheduler.initScheduler() contains a surprising unary plus
>
[jira] [Commented] (YARN-7073) Rest API site documentation
[
https://issues.apache.org/jira/browse/YARN-7073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16148388#comment-16148388
]
Hadoop QA commented on YARN-7073:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
18s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} yarn-native-services Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
21s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
15s{color} | {color:green} yarn-native-services passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 16m
19s{color} | {color:green} yarn-native-services passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
37s{color} | {color:green} yarn-native-services passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
8s{color} | {color:green} yarn-native-services passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
1s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-project hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
29s{color} | {color:green} yarn-native-services passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
51s{color} | {color:green} yarn-native-services passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
20s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 16m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 16m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
45s{color} | {color:green} root: The patch generated 0 new + 4 unchanged - 4
fixed = 4 total (was 8) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}139m
59s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 2 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-project hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
21s{color} | {color:green} hadoop-project in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
26s{color} | {color:green} hadoop-yarn-services-api in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
21s{color} | {color:green} hadoop-yarn-site in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
34s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}226m 27s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | YARN-7073 |
| JIRA Patch URL |
[jira] [Commented] (YARN-7138) Fix incompatible API change for YarnScheduler involved by YARN-5521
[
https://issues.apache.org/jira/browse/YARN-7138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16148381#comment-16148381
]
Wangda Tan commented on YARN-7138:
--
[~djp], APIs of YARN server are private. No class from daemons like RM/NM are
listed in Java API doc:
http://hadoop.apache.org/docs/r3.0.0-alpha4/api/index.html. In the other word,
sub projects for daemons are excluded when we generate javadocs. (Ref:
https://github.com/apache/hadoop/blob/trunk/pom.xml):
{code}
org.apache.hadoop.authentication*,org.apache.hadoop.mapreduce.v2.proto,org.apache.hadoop.yarn.proto,org.apache.hadoop.yarn.server*,org.apache.hadoop.yarn.webapp*
{code}
Even if YARN supports specifying different scheduler implementation other than
Fair/Capacity/Fifo, but we never maintain API compatibility of customized
scheduler.
> Fix incompatible API change for YarnScheduler involved by YARN-5521
> ---
>
> Key: YARN-7138
> URL: https://issues.apache.org/jira/browse/YARN-7138
> Project: Hadoop YARN
> Issue Type: Bug
> Components: scheduler
>Reporter: Junping Du
>Priority: Blocker
>
> From JACC report for 2.8.2 against 2.7.4, it indicates that we have
> incompatible changes happen in YarnScheduler:
> {noformat}
> hadoop-yarn-server-resourcemanager-2.7.4.jar, YarnScheduler.class
> package org.apache.hadoop.yarn.server.resourcemanager.scheduler
> YarnScheduler.allocate ( ApplicationAttemptId p1, List p2,
> List p3, List p4, List p5 ) [abstract] :
> Allocation
> {noformat}
> The root cause is YARN-5221. We should change it back or workaround this by
> adding back original API (mark as deprecated if not used any more).
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-7107) add ability in Fair Scheduler to configure whether disable a queue
[
https://issues.apache.org/jira/browse/YARN-7107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16148376#comment-16148376
]
YunFan Zhou edited comment on YARN-7107 at 8/31/17 3:01 AM:
{noformat}
The trick is how to zero out non-sibling queue without making a mess
{noformat}
Thank [~templedf]. There is no better way to do this for the implementation
does make the code look messy. So, we just have to *zero sibling queue* which I
think it's good enough to make our semantics and behavior more clear.
I think we've basically figured out what we want to achieve, and if there's no
problem, I can start doing it this weekend.
was (Author: daemon):
{noformat}
The trick is how to zero out non-sibling queue without making a mess
{noformat}
Thank [~templedf]. There is no better way to do this, so the implementation
does make the code look messy. So, we just have to zero sibling queue.
I think we've basically figured out what we want to achieve, and if there's no
problem, I can start doing it this weekend.
> add ability in Fair Scheduler to configure whether disable a queue
> --
>
> Key: YARN-7107
> URL: https://issues.apache.org/jira/browse/YARN-7107
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: YunFan Zhou
>Assignee: YunFan Zhou
> Labels: fairscheduler
>
> In a production environment, emergency situations (such as the need to
> calculate the important reports) as soon as possible we need to disable all
> other queues, only allows the *RM* 's resources assigned to emergency queue
> and other queue only at the end of the urgent tasks before allowing them to
> be scheduled properly.
> At present, our approach is to write a script, in the case of an emergency
> manual changes all other queues' *minResources *and *maxResources * to *0mb,
> 0vcores* and then rebase it.This is very troublesome and easy to make
> mistakes.
> So we need to add a configuration in the *FairScheduler* configuration to
> indicate whether the queue is disabled, and if it is disabled, then *RM *will
> not allocate resources to the queue.
> * The child queue will integrate this property of the parent queue.
> * If the child queue is configured with this property, the value of the child
> queue configuration overrides the attributes of the parent queue.
> * The default value of the root queue is *enabled*.
> This will satisfy our needs, and I think other users will encounter such a
> scenario.I think this is very applicable to everyone.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7107) add ability in Fair Scheduler to configure whether disable a queue
[
https://issues.apache.org/jira/browse/YARN-7107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16148376#comment-16148376
]
YunFan Zhou commented on YARN-7107:
---
{noformat}
The trick is how to zero out non-sibling queue without making a mess
{noformat}
Thank [~templedf]. There is no better way to do this, so the implementation
does make the code look messy. So, we just have to zero sibling queue.
I think we've basically figured out what we want to achieve, and if there's no
problem, I can start doing it this weekend.
> add ability in Fair Scheduler to configure whether disable a queue
> --
>
> Key: YARN-7107
> URL: https://issues.apache.org/jira/browse/YARN-7107
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: YunFan Zhou
>Assignee: YunFan Zhou
> Labels: fairscheduler
>
> In a production environment, emergency situations (such as the need to
> calculate the important reports) as soon as possible we need to disable all
> other queues, only allows the *RM* 's resources assigned to emergency queue
> and other queue only at the end of the urgent tasks before allowing them to
> be scheduled properly.
> At present, our approach is to write a script, in the case of an emergency
> manual changes all other queues' *minResources *and *maxResources * to *0mb,
> 0vcores* and then rebase it.This is very troublesome and easy to make
> mistakes.
> So we need to add a configuration in the *FairScheduler* configuration to
> indicate whether the queue is disabled, and if it is disabled, then *RM *will
> not allocate resources to the queue.
> * The child queue will integrate this property of the parent queue.
> * If the child queue is configured with this property, the value of the child
> queue configuration overrides the attributes of the parent queue.
> * The default value of the root queue is *enabled*.
> This will satisfy our needs, and I think other users will encounter such a
> scenario.I think this is very applicable to everyone.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7095) Federation: routing REST invocations transparently to multiple RMs (part 3 - getNodes)
[
https://issues.apache.org/jira/browse/YARN-7095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16148360#comment-16148360
]
Hadoop QA commented on YARN-7095:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
45s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
5s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
38s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
33s{color} | {color:green} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server:
The patch generated 0 new + 18 unchanged - 27 fixed = 18 total (was 45) {color}
|
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 45m 26s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
3s{color} | {color:green} hadoop-yarn-server-router in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 78m 30s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | YARN-7095 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12884574/YARN-7095.v3.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 4d59f1c5865c 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12
13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / ce79f7b |
| Default Java | 1.8.0_144 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-YARN-Build/17222/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/17222/testReport/ |
| modules | C:
[jira] [Updated] (YARN-7132) FairScheduler.initScheduler() contains a surprising unary plus
[
https://issues.apache.org/jira/browse/YARN-7132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yeliang Cang updated YARN-7132:
---
Attachment: YARN-7132.001.patch
Thanks [~templedf]! I have submitted a patch, and please check it!
> FairScheduler.initScheduler() contains a surprising unary plus
> --
>
> Key: YARN-7132
> URL: https://issues.apache.org/jira/browse/YARN-7132
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: fairscheduler
>Affects Versions: 3.0.0-alpha4
>Reporter: Daniel Templeton
>Priority: Minor
> Labels: newbie
> Attachments: YARN-7132.001.patch
>
>
> The method contains the following code:{code}
> LOG.warn(FairSchedulerConfiguration.UPDATE_INTERVAL_MS
> + " is invalid, so using default value "
> + +FairSchedulerConfiguration.DEFAULT_UPDATE_INTERVAL_MS
> + " ms instead");{code}
> Note the beginning of the third line. One of those plusses should be deleted
> so that no one else spends cycles trying to understand why it even compiles.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6964) Fair scheduler misuses Resources operations
[
https://issues.apache.org/jira/browse/YARN-6964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16148342#comment-16148342
]
Hadoop QA commented on YARN-6964:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
14s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
49s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
3s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
17s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
11s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
58s{color} | {color:green} hadoop-yarn-project/hadoop-yarn: The patch generated
0 new + 118 unchanged - 1 fixed = 118 total (was 119) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
10s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
30s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 44m 22s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
30s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}104m 8s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation |
| |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestIncreaseAllocationExpirer
|
| |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerResizing |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | YARN-6964 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12884572/YARN-6964.010.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux d8e8dc768cb6 3.13.0-119-generic #166-Ubuntu SMP Wed May 3
12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 3e0e203 |
| Default Java | 1.8.0_144 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-YARN-Build/17221/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
|
| Test Results |
[jira] [Commented] (YARN-7073) Rest API site documentation
[
https://issues.apache.org/jira/browse/YARN-7073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16148338#comment-16148338
]
Hadoop QA commented on YARN-7073:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 5m
10s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} yarn-native-services Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
11s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
31s{color} | {color:green} yarn-native-services passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m
43s{color} | {color:green} yarn-native-services passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
9s{color} | {color:green} yarn-native-services passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
4s{color} | {color:green} yarn-native-services passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-project hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
30s{color} | {color:green} yarn-native-services passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
57s{color} | {color:green} yarn-native-services passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
18s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 13m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 13m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
5s{color} | {color:green} root: The patch generated 0 new + 3 unchanged - 4
fixed = 3 total (was 7) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}135m
34s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 2 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-project hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
22s{color} | {color:green} hadoop-project in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
28s{color} | {color:green} hadoop-yarn-services-api in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
22s{color} | {color:green} hadoop-yarn-site in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
36s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}219m 29s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | YARN-7073 |
| JIRA Patch URL |
[jira] [Commented] (YARN-7094) Document the current known issue with server-side NM graceful decom
[ https://issues.apache.org/jira/browse/YARN-7094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16148321#comment-16148321 ] Hudson commented on YARN-7094: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #12280 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/12280/]) YARN-7094. Document the current known issue with server-side NM graceful (rkanter: rev ce79f7be29dfbad444614cb340c59e4ff8137e89) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/cli/RMAdminCLI.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/YarnCommands.md > Document the current known issue with server-side NM graceful decom > --- > > Key: YARN-7094 > URL: https://issues.apache.org/jira/browse/YARN-7094 > Project: Hadoop YARN > Issue Type: Sub-task > Components: graceful >Affects Versions: 3.0.0-beta1 >Reporter: Robert Kanter >Assignee: Robert Kanter >Priority: Blocker > Fix For: 2.9.0, 3.0.0-beta1 > > Attachments: YARN-7094.001.patch, YARN-7094.002.patch > > > Server-side NM graceful decom currently does not work correctly when an RM > failover occurs because we don't persist the info in the state store (see > YARN-5464). Given time constraints for Hadoop 3 beta 1, we've decided to > document this limitation and recommend client-side NM graceful decom in the > meantime if you need this functionality (see [this > comment|https://issues.apache.org/jira/browse/YARN-5464?focusedCommentId=16126119=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16126119]). > Once YARN-5464 is done, we can undo this doc change. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7094) Document the current known issue with server-side NM graceful decom
[ https://issues.apache.org/jira/browse/YARN-7094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du updated YARN-7094: - Target Version/s: 2.9.0, 3.0.0-beta1 (was: 3.0.0-beta1) > Document the current known issue with server-side NM graceful decom > --- > > Key: YARN-7094 > URL: https://issues.apache.org/jira/browse/YARN-7094 > Project: Hadoop YARN > Issue Type: Sub-task > Components: graceful >Affects Versions: 3.0.0-beta1 >Reporter: Robert Kanter >Assignee: Robert Kanter >Priority: Blocker > Attachments: YARN-7094.001.patch, YARN-7094.002.patch > > > Server-side NM graceful decom currently does not work correctly when an RM > failover occurs because we don't persist the info in the state store (see > YARN-5464). Given time constraints for Hadoop 3 beta 1, we've decided to > document this limitation and recommend client-side NM graceful decom in the > meantime if you need this functionality (see [this > comment|https://issues.apache.org/jira/browse/YARN-5464?focusedCommentId=16126119=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16126119]). > Once YARN-5464 is done, we can undo this doc change. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5221) Expose UpdateResourceRequest API to allow AM to request for change in container properties
[ https://issues.apache.org/jira/browse/YARN-5221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16148294#comment-16148294 ] Junping Du commented on YARN-5221: -- This patch involve an incompatible change on YarnScheduler#allocate() which is supposed to be public and stable. Just file YARN-7138 to track this. > Expose UpdateResourceRequest API to allow AM to request for change in > container properties > -- > > Key: YARN-5221 > URL: https://issues.apache.org/jira/browse/YARN-5221 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Arun Suresh >Assignee: Arun Suresh > Fix For: 2.8.0, 3.0.0-alpha2 > > Attachments: YARN-5221.001.patch, YARN-5221.002.patch, > YARN-5221.003.patch, YARN-5221.004.patch, YARN-5221.005.patch, > YARN-5221.006.patch, YARN-5221.007.patch, YARN-5221.008.patch, > YARN-5221.009.patch, YARN-5221.010.patch, YARN-5221.011.patch, > YARN-5221.012.patch, YARN-5221.013.patch, YARN-5221-branch-2.8-v1.patch, > YARN-5221-branch-2-v1.patch > > > YARN-1197 introduced APIs to allow an AM to request for Increase and Decrease > of Container Resources after initial allocation. > YARN-5085 proposes to allow an AM to request for a change of Container > ExecutionType. > This JIRA proposes to unify both of the above into an Update Container API. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7138) Fix incompatible API change for YarnScheduler involved by YARN-5521
[
https://issues.apache.org/jira/browse/YARN-7138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16148295#comment-16148295
]
Junping Du commented on YARN-7138:
--
CC [~leftnoteasy], [~asuresh].
> Fix incompatible API change for YarnScheduler involved by YARN-5521
> ---
>
> Key: YARN-7138
> URL: https://issues.apache.org/jira/browse/YARN-7138
> Project: Hadoop YARN
> Issue Type: Bug
> Components: scheduler
>Reporter: Junping Du
>Priority: Blocker
>
> From JACC report for 2.8.2 against 2.7.4, it indicates that we have
> incompatible changes happen in YarnScheduler:
> {noformat}
> hadoop-yarn-server-resourcemanager-2.7.4.jar, YarnScheduler.class
> package org.apache.hadoop.yarn.server.resourcemanager.scheduler
> YarnScheduler.allocate ( ApplicationAttemptId p1, List p2,
> List p3, List p4, List p5 ) [abstract] :
> Allocation
> {noformat}
> The root cause is YARN-5221. We should change it back or workaround this by
> adding back original API (mark as deprecated if not used any more).
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-7138) Fix incompatible API change for YarnScheduler involved by YARN-5521
Junping Du created YARN-7138:
Summary: Fix incompatible API change for YarnScheduler involved by
YARN-5521
Key: YARN-7138
URL: https://issues.apache.org/jira/browse/YARN-7138
Project: Hadoop YARN
Issue Type: Bug
Components: scheduler
Reporter: Junping Du
Priority: Blocker
>From JACC report for 2.8.2 against 2.7.4, it indicates that we have
>incompatible changes happen in YarnScheduler:
{noformat}
hadoop-yarn-server-resourcemanager-2.7.4.jar, YarnScheduler.class
package org.apache.hadoop.yarn.server.resourcemanager.scheduler
YarnScheduler.allocate ( ApplicationAttemptId p1, List p2,
List p3, List p4, List p5 ) [abstract] :
Allocation
{noformat}
The root cause is YARN-5221. We should change it back or workaround this by
adding back original API (mark as deprecated if not used any more).
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7094) Document the current known issue with server-side NM graceful decom
[ https://issues.apache.org/jira/browse/YARN-7094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Kanter updated YARN-7094: Summary: Document the current known issue with server-side NM graceful decom (was: Document the current known issue with server-side graceful decom) > Document the current known issue with server-side NM graceful decom > --- > > Key: YARN-7094 > URL: https://issues.apache.org/jira/browse/YARN-7094 > Project: Hadoop YARN > Issue Type: Sub-task > Components: graceful >Affects Versions: 3.0.0-beta1 >Reporter: Robert Kanter >Assignee: Robert Kanter >Priority: Blocker > Attachments: YARN-7094.001.patch, YARN-7094.002.patch > > > Server-side NM graceful decom currently does not work correctly when an RM > failover occurs because we don't persist the info in the state store (see > YARN-5464). Given time constraints for Hadoop 3 beta 1, we've decided to > document this limitation and recommend client-side NM graceful decom in the > meantime if you need this functionality (see [this > comment|https://issues.apache.org/jira/browse/YARN-5464?focusedCommentId=16126119=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16126119]). > Once YARN-5464 is done, we can undo this doc change. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7094) Document the current known issue with server-side graceful decom
[ https://issues.apache.org/jira/browse/YARN-7094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Kanter updated YARN-7094: Summary: Document the current known issue with server-side graceful decom (was: Document the current limitation of server-side graceful decom) > Document the current known issue with server-side graceful decom > > > Key: YARN-7094 > URL: https://issues.apache.org/jira/browse/YARN-7094 > Project: Hadoop YARN > Issue Type: Sub-task > Components: graceful >Affects Versions: 3.0.0-beta1 >Reporter: Robert Kanter >Assignee: Robert Kanter >Priority: Blocker > Attachments: YARN-7094.001.patch, YARN-7094.002.patch > > > Server-side NM graceful decom currently does not work correctly when an RM > failover occurs because we don't persist the info in the state store (see > YARN-5464). Given time constraints for Hadoop 3 beta 1, we've decided to > document this limitation and recommend client-side NM graceful decom in the > meantime if you need this functionality (see [this > comment|https://issues.apache.org/jira/browse/YARN-5464?focusedCommentId=16126119=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16126119]). > Once YARN-5464 is done, we can undo this doc change. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7094) Document the current limitation of server-side graceful decom
[ https://issues.apache.org/jira/browse/YARN-7094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Kanter updated YARN-7094: Summary: Document the current limitation of server-side graceful decom (was: Document that server-side graceful decom is currently not recommended) > Document the current limitation of server-side graceful decom > - > > Key: YARN-7094 > URL: https://issues.apache.org/jira/browse/YARN-7094 > Project: Hadoop YARN > Issue Type: Sub-task > Components: graceful >Affects Versions: 3.0.0-beta1 >Reporter: Robert Kanter >Assignee: Robert Kanter >Priority: Blocker > Attachments: YARN-7094.001.patch, YARN-7094.002.patch > > > Server-side NM graceful decom currently does not work correctly when an RM > failover occurs because we don't persist the info in the state store (see > YARN-5464). Given time constraints for Hadoop 3 beta 1, we've decided to > document this limitation and recommend client-side NM graceful decom in the > meantime if you need this functionality (see [this > comment|https://issues.apache.org/jira/browse/YARN-5464?focusedCommentId=16126119=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16126119]). > Once YARN-5464 is done, we can undo this doc change. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7136) Additional Performance Improvement for Resource Profile Feature
[
https://issues.apache.org/jira/browse/YARN-7136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16148289#comment-16148289
]
Hadoop QA commented on YARN-7136:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 3m
24s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} YARN-3926 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
57s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
25s{color} | {color:green} YARN-3926 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m
18s{color} | {color:green} YARN-3926 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
56s{color} | {color:green} YARN-3926 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
21s{color} | {color:green} YARN-3926 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
49s{color} | {color:green} YARN-3926 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
14s{color} | {color:green} YARN-3926 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m
24s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 55s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch
generated 16 new + 13 unchanged - 2 fixed = 29 total (was 15) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
29s{color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
10s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
39s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 2m 47s{color}
| {color:red} hadoop-yarn-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
32s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 62m 45s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api |
| | org.apache.hadoop.yarn.api.records.Resource.getResources() may expose
internal representation by returning Resource.resources At Resource.java:by
returning Resource.resources At Resource.java:[line 214] |
| Failed junit tests | hadoop.yarn.util.resource.TestResourceCalculator |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | YARN-7136 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12884570/YARN-7136.YARN-3926.002.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 9d93c6f08125 3.13.0-123-generic #172-Ubuntu SMP Mon Jun 26
18:04:35 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git
[jira] [Commented] (YARN-7095) Federation: routing REST invocations transparently to multiple RMs (part 3 - getNodes)
[
https://issues.apache.org/jira/browse/YARN-7095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16148287#comment-16148287
]
Giovanni Matteo Fumarola commented on YARN-7095:
The merge logic is fully validate in {{TestRouterWebServiceUtil}}. In
{{TestFederationInterceptorREST}} and {{TestFederationInterceptorRESTRetry}} I
only validate the correctness of the interceptor. I added some comments around
in v3.
> Federation: routing REST invocations transparently to multiple RMs (part 3 -
> getNodes)
> --
>
> Key: YARN-7095
> URL: https://issues.apache.org/jira/browse/YARN-7095
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Giovanni Matteo Fumarola
>Assignee: Giovanni Matteo Fumarola
> Attachments: YARN-7095.v1-rebase.patch, YARN-7095.v2.patch,
> YARN-7095.v3.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7095) Federation: routing REST invocations transparently to multiple RMs (part 3 - getNodes)
[ https://issues.apache.org/jira/browse/YARN-7095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Giovanni Matteo Fumarola updated YARN-7095: --- Attachment: YARN-7095.v3.patch > Federation: routing REST invocations transparently to multiple RMs (part 3 - > getNodes) > -- > > Key: YARN-7095 > URL: https://issues.apache.org/jira/browse/YARN-7095 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Giovanni Matteo Fumarola >Assignee: Giovanni Matteo Fumarola > Attachments: YARN-7095.v1-rebase.patch, YARN-7095.v2.patch, > YARN-7095.v3.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7115) Move BoundedAppender to org.hadoop.yarn.util pacakge
[ https://issues.apache.org/jira/browse/YARN-7115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16148281#comment-16148281 ] Hudson commented on YARN-7115: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #12279 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/12279/]) YARN-7115. Move BoundedAppender to org.hadoop.yarn.util pacakge (templedf: rev cc23514abacb4d6589e731cc5ce5d8e6f19c955d) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/RMAppAttemptImpl.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/BoundedAppender.java * (delete) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/TestBoundedAppender.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestBoundedAppender.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/pom.xml * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/TestRMAppAttemptImplDiagnostics.java > Move BoundedAppender to org.hadoop.yarn.util pacakge > - > > Key: YARN-7115 > URL: https://issues.apache.org/jira/browse/YARN-7115 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 3.0.0-alpha4 >Reporter: Jian He >Assignee: Jian He > Fix For: 3.0.0-beta1 > > Attachments: YARN-7115.01.patch, YARN-7115.02.patch, > YARN-7115.03.patch, YARN-7115.04.patch > > > BoundedAppender is a useful util class which can be present in the util > package -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7095) Federation: routing REST invocations transparently to multiple RMs (part 3 - getNodes)
[ https://issues.apache.org/jira/browse/YARN-7095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16148277#comment-16148277 ] Subru Krishnan commented on YARN-7095: -- Thanks [~giovanni.fumarola] for updating the patch. I agree that including *getClusterMetricsInfo* also makes sense as it follows an identical pattern. Other than Yetus warnings, I have only one comment about the tests: * Can you check for all (or as many as you can) attributes of nodes and cluster metrics when you are asserting so that we can fully validate the merge logic. I prefer a helper method which you consistently use in all tests. > Federation: routing REST invocations transparently to multiple RMs (part 3 - > getNodes) > -- > > Key: YARN-7095 > URL: https://issues.apache.org/jira/browse/YARN-7095 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Giovanni Matteo Fumarola >Assignee: Giovanni Matteo Fumarola > Attachments: YARN-7095.v1-rebase.patch, YARN-7095.v2.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7095) Federation: routing REST invocations transparently to multiple RMs (part 3 - getNodes)
[
https://issues.apache.org/jira/browse/YARN-7095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16148266#comment-16148266
]
Giovanni Matteo Fumarola commented on YARN-7095:
By mistake in v2 there is a formatting for {{ContainersLauncher}} and the
findbug warning is related to it. I will remove it in the next iteration.
> Federation: routing REST invocations transparently to multiple RMs (part 3 -
> getNodes)
> --
>
> Key: YARN-7095
> URL: https://issues.apache.org/jira/browse/YARN-7095
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Giovanni Matteo Fumarola
>Assignee: Giovanni Matteo Fumarola
> Attachments: YARN-7095.v1-rebase.patch, YARN-7095.v2.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7095) Federation: routing REST invocations transparently to multiple RMs (part 3 - getNodes)
[
https://issues.apache.org/jira/browse/YARN-7095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16148261#comment-16148261
]
Hadoop QA commented on YARN-7095:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
24s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
8s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
40s{color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
51s{color} | {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
in trunk has 1 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
0s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s{color} | {color:green} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server:
The patch generated 0 new + 65 unchanged - 27 fixed = 65 total (was 92) {color}
|
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 14m
9s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed.
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 47m 13s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
12s{color} | {color:green} hadoop-yarn-server-router in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}102m 13s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation |
| | hadoop.yarn.server.resourcemanager.scheduler.fair.TestFSAppStarvation |
| Timed out junit tests |
org.apache.hadoop.yarn.server.resourcemanager.TestSubmitApplicationWithRMHA |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | YARN-7095 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12884561/YARN-7095.v2.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux d3a9733c7666 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31
14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision |
[jira] [Assigned] (YARN-1014) Configure OOM Killer to kill OPPORTUNISTIC containers first
[ https://issues.apache.org/jira/browse/YARN-1014?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haibo Chen reassigned YARN-1014: Assignee: Haibo Chen (was: Karthik Kambatla) > Configure OOM Killer to kill OPPORTUNISTIC containers first > --- > > Key: YARN-1014 > URL: https://issues.apache.org/jira/browse/YARN-1014 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 3.0.0-alpha1 >Reporter: Arun C Murthy >Assignee: Haibo Chen > Attachments: YARN-1014.00.patch, YARN-1014.01.patch, > YARN-1014.02.patch > > > YARN-2882 introduces the notion of OPPORTUNISTIC containers. These containers > should be killed first should the system run out of memory. > - > Previous description: > Once RM allocates 'speculative containers' we need to get LCE to schedule > them at lower priorities via cgroups. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7115) Move BoundedAppender to org.hadoop.yarn.util pacakge
[ https://issues.apache.org/jira/browse/YARN-7115?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Templeton updated YARN-7115: --- Affects Version/s: 3.0.0-alpha4 > Move BoundedAppender to org.hadoop.yarn.util pacakge > - > > Key: YARN-7115 > URL: https://issues.apache.org/jira/browse/YARN-7115 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 3.0.0-alpha4 >Reporter: Jian He >Assignee: Jian He > Fix For: 3.0.0-beta1 > > Attachments: YARN-7115.01.patch, YARN-7115.02.patch, > YARN-7115.03.patch, YARN-7115.04.patch > > > BoundedAppender is a useful util class which can be present in the util > package -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6964) Fair scheduler misuses Resources operations
[
https://issues.apache.org/jira/browse/YARN-6964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Templeton updated YARN-6964:
---
Attachment: YARN-6964.010.patch
Good point. New patch without the method.
> Fair scheduler misuses Resources operations
> ---
>
> Key: YARN-6964
> URL: https://issues.apache.org/jira/browse/YARN-6964
> Project: Hadoop YARN
> Issue Type: Bug
> Components: fairscheduler
>Affects Versions: 3.0.0-alpha4
>Reporter: Daniel Templeton
>Assignee: Daniel Templeton
> Attachments: YARN-6964.001.patch, YARN-6964.002.patch,
> YARN-6964.003.patch, YARN-6964.004.patch, YARN-6964.005.patch,
> YARN-6964.006.patch, YARN-6964.007.patch, YARN-6964.008.patch,
> YARN-6964.009.patch, YARN-6964.010.patch
>
>
> There are several places where YARN uses the {{Resources}} class to do
> comparisons of {{Resource}} instances incorrectly. This patch corrects those
> mistakes.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7123) FairScheduler.getResourceCalculator() returns an instance of DefaultResourceCalculator regardless of the configuration
[
https://issues.apache.org/jira/browse/YARN-7123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16148246#comment-16148246
]
Daniel Templeton commented on YARN-7123:
Per offline discussion with [~kasha], it really is an issue, and we don't need
it to be configurable. Let's just make the default be a
{{DominantResourceCalculator}} instance.
> FairScheduler.getResourceCalculator() returns an instance of
> DefaultResourceCalculator regardless of the configuration
> --
>
> Key: YARN-7123
> URL: https://issues.apache.org/jira/browse/YARN-7123
> Project: Hadoop YARN
> Issue Type: Bug
> Components: fairscheduler
>Affects Versions: 3.0.0-alpha4
>Reporter: Daniel Templeton
> Attachments: YARN-7123.branch-3.0.0-alpha4.001.patch
>
>
> There are several places where this creates the wrong behavior:
> * 298:RMServerUtils.java
> * 1081:AbstractYarnScheduler.java
> * 1197:FSAppAttempt.java
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7013) merge related work for YARN-3926 branch
[ https://issues.apache.org/jira/browse/YARN-7013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-7013: - Attachment: YARN-7013.006.patch Attached ver.6 patch, included ver.2 patch from YARN-7136 > merge related work for YARN-3926 branch > --- > > Key: YARN-7013 > URL: https://issues.apache.org/jira/browse/YARN-7013 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Sunil G >Assignee: Sunil G > Attachments: YARN-7013.001.patch, YARN-7013.002.patch, > YARN-7013.003.patch, YARN-7013.004.patch, YARN-7013.005.patch, > YARN-7013.006.patch > > > To run jenkins for whole branch. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7136) Additional Performance Improvement for Resource Profile Feature
[ https://issues.apache.org/jira/browse/YARN-7136?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-7136: - Attachment: YARN-7136.YARN-3926.002.patch Attached ver.2 patch, fixed findbugs warning / javadocs warning. Also updated ResourcePBImpl inherit from Resource instead of BaseResource class. I found this change will be clear for method overwrite. And it's better to decouple ResourcePBImpl from BaseResource, I don't see any benefit from inheriting BaseResource for -PBImpl > Additional Performance Improvement for Resource Profile Feature > --- > > Key: YARN-7136 > URL: https://issues.apache.org/jira/browse/YARN-7136 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Wangda Tan >Assignee: Wangda Tan >Priority: Critical > Attachments: YARN-7136.001.patch, YARN-7136.YARN-3926.001.patch, > YARN-7136.YARN-3926.002.patch > > > This JIRA is plan to add following misc perf improvements: > 1) Use final int in Resources/ResourceCalculator to cache > #known-resource-types. (Significant improvement). > 2) Catch Java's ArrayOutOfBound Exception instead of checking array.length > every time. (Significant improvement). > 3) Avoid setUnit validation (which is a HashSet lookup) when initialize > default Memory/VCores ResourceInformation (Significant improvement). > 4) Avoid unnecessary loop array in Resource#toString/hashCode. (Some > improvement). > 5) Removed readOnlyResources in BaseResource. (Minor improvement). > 6) Removed enum: MandatoryResources, use final integer instead. (Minor > improvement). -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7136) Additional Performance Improvement for Resource Profile Feature
[
https://issues.apache.org/jira/browse/YARN-7136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16148216#comment-16148216
]
Hadoop QA commented on YARN-7136:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} YARN-3926 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
47s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
51s{color} | {color:green} YARN-3926 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m
30s{color} | {color:green} YARN-3926 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
5s{color} | {color:green} YARN-3926 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
30s{color} | {color:green} YARN-3926 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
31s{color} | {color:green} YARN-3926 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
27s{color} | {color:green} YARN-3926 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
14s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m
33s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 6s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch
generated 11 new + 15 unchanged - 1 fixed = 26 total (was 16) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
49s{color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
25s{color} | {color:red} hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api
generated 1 new + 123 unchanged - 0 fixed = 124 total (was 123) {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
34s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 3m 0s{color}
| {color:red} hadoop-yarn-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
28s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 68m 16s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api |
| | org.apache.hadoop.yarn.api.records.Resource defines equals and uses
Object.hashCode() At Resource.java:Object.hashCode() At Resource.java:[lines
378-412] |
| Failed junit tests | hadoop.yarn.util.resource.TestResourceCalculator |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | YARN-7136 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12884555/YARN-7136.YARN-3926.001.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 198c82166283 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31
14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
[jira] [Commented] (YARN-6964) Fair scheduler misuses Resources operations
[
https://issues.apache.org/jira/browse/YARN-6964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16148212#comment-16148212
]
Karthik Kambatla commented on YARN-6964:
Taking a fresh look. Only one comment: do we need isNonZero. Can we not just
use {{!isAnyMajorZero}}?
> Fair scheduler misuses Resources operations
> ---
>
> Key: YARN-6964
> URL: https://issues.apache.org/jira/browse/YARN-6964
> Project: Hadoop YARN
> Issue Type: Bug
> Components: fairscheduler
>Affects Versions: 3.0.0-alpha4
>Reporter: Daniel Templeton
>Assignee: Daniel Templeton
> Attachments: YARN-6964.001.patch, YARN-6964.002.patch,
> YARN-6964.003.patch, YARN-6964.004.patch, YARN-6964.005.patch,
> YARN-6964.006.patch, YARN-6964.007.patch, YARN-6964.008.patch,
> YARN-6964.009.patch
>
>
> There are several places where YARN uses the {{Resources}} class to do
> comparisons of {{Resource}} instances incorrectly. This patch corrects those
> mistakes.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7013) merge related work for YARN-3926 branch
[ https://issues.apache.org/jira/browse/YARN-7013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-7013: - Attachment: YARN-7013.005.patch > merge related work for YARN-3926 branch > --- > > Key: YARN-7013 > URL: https://issues.apache.org/jira/browse/YARN-7013 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Sunil G >Assignee: Sunil G > Attachments: YARN-7013.001.patch, YARN-7013.002.patch, > YARN-7013.003.patch, YARN-7013.004.patch, YARN-7013.005.patch > > > To run jenkins for whole branch. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7073) Rest API site documentation
[ https://issues.apache.org/jira/browse/YARN-7073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jian He updated YARN-7073: -- Attachment: YARN-7073-yarn-native-services.005.patch > Rest API site documentation > --- > > Key: YARN-7073 > URL: https://issues.apache.org/jira/browse/YARN-7073 > Project: Hadoop YARN > Issue Type: Sub-task > Components: documentation, site >Reporter: Gour Saha >Assignee: Gour Saha > Attachments: YARN-7073-yarn-native-services.001.patch, > YARN-7073-yarn-native-services.002.patch, > YARN-7073-yarn-native-services.003.patch, > YARN-7073-yarn-native-services.004.patch, > YARN-7073-yarn-native-services.005.patch > > > Commit site documentation for REST API service, generated from the swagger > definition as a MD file. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7137) Resource profile: Move new added APIs to unstable
[ https://issues.apache.org/jira/browse/YARN-7137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-7137: - Attachment: YARN-7137.YARN-3926.001.patch Uploaded ver.001 patch, this patch is on top of YARN-7136. > Resource profile: Move new added APIs to unstable > - > > Key: YARN-7137 > URL: https://issues.apache.org/jira/browse/YARN-7137 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Wangda Tan >Assignee: Wangda Tan >Priority: Blocker > Attachments: YARN-7137.YARN-3926.001.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-7137) Resource profile: Move new added APIs to unstable
Wangda Tan created YARN-7137: Summary: Resource profile: Move new added APIs to unstable Key: YARN-7137 URL: https://issues.apache.org/jira/browse/YARN-7137 Project: Hadoop YARN Issue Type: Sub-task Reporter: Wangda Tan Assignee: Wangda Tan Priority: Blocker -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7095) Federation: routing REST invocations transparently to multiple RMs (part 3 - getNodes)
[
https://issues.apache.org/jira/browse/YARN-7095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16148180#comment-16148180
]
Giovanni Matteo Fumarola commented on YARN-7095:
Thanks [~subru] for the feedback. I fixed (1) and (2) but we cannot apply your
feedback (3) since the {{NodeInfo}} have the same NodeId but a different
HealthReport and we need to pick the newest one.
In patch v2 I added also the code for GetClusterMetrics, we can close YARN-7096
or rename it.
> Federation: routing REST invocations transparently to multiple RMs (part 3 -
> getNodes)
> --
>
> Key: YARN-7095
> URL: https://issues.apache.org/jira/browse/YARN-7095
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Giovanni Matteo Fumarola
>Assignee: Giovanni Matteo Fumarola
> Attachments: YARN-7095.v1-rebase.patch, YARN-7095.v2.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7095) Federation: routing REST invocations transparently to multiple RMs (part 3 - getNodes)
[ https://issues.apache.org/jira/browse/YARN-7095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Giovanni Matteo Fumarola updated YARN-7095: --- Attachment: YARN-7095.v2.patch > Federation: routing REST invocations transparently to multiple RMs (part 3 - > getNodes) > -- > > Key: YARN-7095 > URL: https://issues.apache.org/jira/browse/YARN-7095 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Giovanni Matteo Fumarola >Assignee: Giovanni Matteo Fumarola > Attachments: YARN-7095.v1-rebase.patch, YARN-7095.v2.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-2162) add ability in Fair Scheduler to optionally configure maxResources in terms of percentage
[ https://issues.apache.org/jira/browse/YARN-2162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16148158#comment-16148158 ] Wangda Tan commented on YARN-2162: -- Thanks [~yufeigu] for explanations! > add ability in Fair Scheduler to optionally configure maxResources in terms > of percentage > - > > Key: YARN-2162 > URL: https://issues.apache.org/jira/browse/YARN-2162 > Project: Hadoop YARN > Issue Type: Improvement > Components: fairscheduler, scheduler >Reporter: Ashwin Shankar >Assignee: Yufei Gu > Labels: scheduler > Attachments: YARN-2162.001.patch, YARN-2162.002.patch, > YARN-2162.003.patch > > > minResources and maxResources in fair scheduler configs are expressed in > terms of absolute numbers X mb, Y vcores. > As a result, when we expand or shrink our hadoop cluster, we need to > recalculate and change minResources/maxResources accordingly, which is pretty > inconvenient. > We can circumvent this problem if we can optionally configure these > properties in terms of percentage of cluster capacity. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7117) Capacity Scheduler: Support Auto Creation of Leaf Queues While Doing Queue Mapping
[
https://issues.apache.org/jira/browse/YARN-7117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16148148#comment-16148148
]
Wangda Tan commented on YARN-7117:
--
Thans [~jlowe], make sense.
[~curino], thanks for pointing me the ReservationSystem related implementation,
I don't want to reinvent the wheel, will definitely investigate how can we
reuse the logic.
> Capacity Scheduler: Support Auto Creation of Leaf Queues While Doing Queue
> Mapping
> --
>
> Key: YARN-7117
> URL: https://issues.apache.org/jira/browse/YARN-7117
> Project: Hadoop YARN
> Issue Type: New Feature
> Components: capacity scheduler
>Reporter: Wangda Tan
>Assignee: Wangda Tan
>
> Currently Capacity Scheduler doesn't support auto creation of queues when
> doing queue mapping. We saw more and more use cases which has complex queue
> mapping policies configured to handle application to queues mapping.
> The most common use case of CapacityScheduler queue mapping is to create one
> queue for each user/group. However update {{capacity-scheduler.xml}} and
> {{RMAdmin:refreshQueues}} needs to be done when new user/group onboard. One
> of the option to solve the problem is automatically create queues when new
> user/group arrives.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6686) Support for adding and removing queue mappings
[
https://issues.apache.org/jira/browse/YARN-6686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16148147#comment-16148147
]
Jonathan Hung commented on YARN-6686:
-
Hi [~csexz], yes, this is a valid concern. Actually I discussed this previously
with Wangda. We will likely have to find a slightly different approach.
> Support for adding and removing queue mappings
> --
>
> Key: YARN-6686
> URL: https://issues.apache.org/jira/browse/YARN-6686
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Jonathan Hung
>Assignee: Jonathan Hung
> Attachments: YARN-6686-YARN-5734.001.patch
>
>
> Right now capacity scheduler uses UserGroupMappingPlacementRule to determine
> queue mappings. This rule stores mappings in
> {{yarn.scheduler.capacity.queue-mappings}}. For users with a large number of
> mappings, adding or removing queue mappings becomes infeasible.
> Need to come up with a way to add/remove individual mappings, for any/all
> different configured placement rules.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7136) Additional Performance Improvement for Resource Profile Feature
[ https://issues.apache.org/jira/browse/YARN-7136?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-7136: - Attachment: YARN-7136.YARN-3926.001.patch Rename the patch to run against YARN-3926 branch. > Additional Performance Improvement for Resource Profile Feature > --- > > Key: YARN-7136 > URL: https://issues.apache.org/jira/browse/YARN-7136 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Wangda Tan >Assignee: Wangda Tan >Priority: Critical > Attachments: YARN-7136.001.patch, YARN-7136.YARN-3926.001.patch > > > This JIRA is plan to add following misc perf improvements: > 1) Use final int in Resources/ResourceCalculator to cache > #known-resource-types. (Significant improvement). > 2) Catch Java's ArrayOutOfBound Exception instead of checking array.length > every time. (Significant improvement). > 3) Avoid setUnit validation (which is a HashSet lookup) when initialize > default Memory/VCores ResourceInformation (Significant improvement). > 4) Avoid unnecessary loop array in Resource#toString/hashCode. (Some > improvement). > 5) Removed readOnlyResources in BaseResource. (Minor improvement). > 6) Removed enum: MandatoryResources, use final integer instead. (Minor > improvement). -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7136) Additional Performance Improvement for Resource Profile Feature
[
https://issues.apache.org/jira/browse/YARN-7136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16148131#comment-16148131
]
Hadoop QA commented on YARN-7136:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 5s{color}
| {color:red} YARN-7136 does not apply to trunk. Rebase required? Wrong Branch?
See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | YARN-7136 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12884553/YARN-7136.001.patch |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/17213/console |
| Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Additional Performance Improvement for Resource Profile Feature
> ---
>
> Key: YARN-7136
> URL: https://issues.apache.org/jira/browse/YARN-7136
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager, resourcemanager
>Reporter: Wangda Tan
>Assignee: Wangda Tan
>Priority: Critical
> Attachments: YARN-7136.001.patch
>
>
> This JIRA is plan to add following misc perf improvements:
> 1) Use final int in Resources/ResourceCalculator to cache
> #known-resource-types. (Significant improvement).
> 2) Catch Java's ArrayOutOfBound Exception instead of checking array.length
> every time. (Significant improvement).
> 3) Avoid setUnit validation (which is a HashSet lookup) when initialize
> default Memory/VCores ResourceInformation (Significant improvement).
> 4) Avoid unnecessary loop array in Resource#toString/hashCode. (Some
> improvement).
> 5) Removed readOnlyResources in BaseResource. (Minor improvement).
> 6) Removed enum: MandatoryResources, use final integer instead. (Minor
> improvement).
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7073) Rest API site documentation
[ https://issues.apache.org/jira/browse/YARN-7073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jian He updated YARN-7073: -- Attachment: YARN-7073-yarn-native-services.004.patch > Rest API site documentation > --- > > Key: YARN-7073 > URL: https://issues.apache.org/jira/browse/YARN-7073 > Project: Hadoop YARN > Issue Type: Sub-task > Components: documentation, site >Reporter: Gour Saha >Assignee: Gour Saha > Attachments: YARN-7073-yarn-native-services.001.patch, > YARN-7073-yarn-native-services.002.patch, > YARN-7073-yarn-native-services.003.patch, > YARN-7073-yarn-native-services.004.patch > > > Commit site documentation for REST API service, generated from the swagger > definition as a MD file. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7136) Additional Performance Improvement for Resource Profile Feature
[ https://issues.apache.org/jira/browse/YARN-7136?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-7136: - Attachment: YARN-7136.001.patch Credit to [~jlowe]/[~sunilg] for offline suggestions, helps to locate issues and providing patches. Consolidated offline suggestions / patches and upload patch ver.001 for review. >From my local test, now comparing YARN-3926 and trunk, there's about 1% >performance regression. > Additional Performance Improvement for Resource Profile Feature > --- > > Key: YARN-7136 > URL: https://issues.apache.org/jira/browse/YARN-7136 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Wangda Tan >Assignee: Wangda Tan >Priority: Critical > Attachments: YARN-7136.001.patch > > > This JIRA is plan to add following misc perf improvements: > 1) Use final int in Resources/ResourceCalculator to cache > #known-resource-types. (Significant improvement). > 2) Catch Java's ArrayOutOfBound Exception instead of checking array.length > every time. (Significant improvement). > 3) Avoid setUnit validation (which is a HashSet lookup) when initialize > default Memory/VCores ResourceInformation (Significant improvement). > 4) Avoid unnecessary loop array in Resource#toString/hashCode. (Some > improvement). > 5) Removed readOnlyResources in BaseResource. (Minor improvement). > 6) Removed enum: MandatoryResources, use final integer instead. (Minor > improvement). -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7136) Additional Performance Improvement for Resource Profile Feature
[ https://issues.apache.org/jira/browse/YARN-7136?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-7136: - Target Version/s: YARN-3926 > Additional Performance Improvement for Resource Profile Feature > --- > > Key: YARN-7136 > URL: https://issues.apache.org/jira/browse/YARN-7136 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Wangda Tan >Assignee: Wangda Tan >Priority: Critical > > This JIRA is plan to add following misc perf improvements: > 1) Use final int in Resources/ResourceCalculator to cache > #known-resource-types. (Significant improvement). > 2) Catch Java's ArrayOutOfBound Exception instead of checking array.length > every time. (Significant improvement). > 3) Avoid setUnit validation (which is a HashSet lookup) when initialize > default Memory/VCores ResourceInformation (Significant improvement). > 4) Avoid unnecessary loop array in Resource#toString/hashCode. (Some > improvement). > 5) Removed readOnlyResources in BaseResource. (Minor improvement). > 6) Removed enum: MandatoryResources, use final integer instead. (Minor > improvement). -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-7136) Additional Performance Improvement for Resource Profile Feature
Wangda Tan created YARN-7136: Summary: Additional Performance Improvement for Resource Profile Feature Key: YARN-7136 URL: https://issues.apache.org/jira/browse/YARN-7136 Project: Hadoop YARN Issue Type: Sub-task Reporter: Wangda Tan Assignee: Wangda Tan Priority: Critical This JIRA is plan to add following misc perf improvements: 1) Use final int in Resources/ResourceCalculator to cache #known-resource-types. (Significant improvement). 2) Catch Java's ArrayOutOfBound Exception instead of checking array.length every time. (Significant improvement). 3) Avoid setUnit validation (which is a HashSet lookup) when initialize default Memory/VCores ResourceInformation (Significant improvement). 4) Avoid unnecessary loop array in Resource#toString/hashCode. (Some improvement). 5) Removed readOnlyResources in BaseResource. (Minor improvement). 6) Removed enum: MandatoryResources, use final integer instead. (Minor improvement). -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-7135) Clean up lock-try order in common scheduler code
Daniel Templeton created YARN-7135:
--
Summary: Clean up lock-try order in common scheduler code
Key: YARN-7135
URL: https://issues.apache.org/jira/browse/YARN-7135
Project: Hadoop YARN
Issue Type: Improvement
Components: scheduler
Affects Versions: 3.0.0-alpha4
Reporter: Daniel Templeton
There are many places that follow the pattern:{code}try {
lock.lock();
...
} finally {
lock.unlock();
}{code}
There are a couple of reasons that's a bad idea. The correct pattern
is:{code}lock.lock();
try {
...
} finally {
lock.unlock();
}{code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7133) Clean up lock-try order in fair scheduler
[
https://issues.apache.org/jira/browse/YARN-7133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Templeton updated YARN-7133:
---
Summary: Clean up lock-try order in fair scheduler (was: Clean up lock-try
order fair scheduler)
> Clean up lock-try order in fair scheduler
> -
>
> Key: YARN-7133
> URL: https://issues.apache.org/jira/browse/YARN-7133
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: fairscheduler
>Affects Versions: 3.0.0-alpha4
>Reporter: Daniel Templeton
>Assignee: Daniel Templeton
> Labels: newbie
>
> There are many places that follow the pattern:{code}try {
> lock.lock();
> ...
> } finally {
> lock.unlock();
> }{code}
> There are a couple of reasons that's a bad idea. The correct pattern
> is:{code}lock.lock();
> try {
> ...
> } finally {
> lock.unlock();
> }{code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-7133) Clean up lock-try order fair scheduler
[
https://issues.apache.org/jira/browse/YARN-7133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Templeton reassigned YARN-7133:
--
Assignee: Daniel Templeton
> Clean up lock-try order fair scheduler
> --
>
> Key: YARN-7133
> URL: https://issues.apache.org/jira/browse/YARN-7133
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: fairscheduler
>Affects Versions: 3.0.0-alpha4
>Reporter: Daniel Templeton
>Assignee: Daniel Templeton
> Labels: newbie
>
> There are many places that follow the pattern:{code}try {
> lock.lock();
> ...
> } finally {
> lock.unlock();
> }{code}
> There are a couple of reasons that's a bad idea. The correct pattern
> is:{code}lock.lock();
> try {
> ...
> } finally {
> lock.unlock();
> }{code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-7134) AppSchedulingInfo has a dependency on capacity scheduler
Daniel Templeton created YARN-7134:
--
Summary: AppSchedulingInfo has a dependency on capacity scheduler
Key: YARN-7134
URL: https://issues.apache.org/jira/browse/YARN-7134
Project: Hadoop YARN
Issue Type: Bug
Components: scheduler
Affects Versions: 3.0.0-alpha4
Reporter: Daniel Templeton
Priority: Blocker
The common scheduling code should be independent of all scheduler
implementations. YARN-6040 introduced capacity scheduler's {{SchedulingMode}}
into {{AppSchedulingInfo}}.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7132) FairScheduler.initScheduler() contains a surprising unary plus
[
https://issues.apache.org/jira/browse/YARN-7132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Templeton updated YARN-7132:
---
Labels: newbie (was: )
> FairScheduler.initScheduler() contains a surprising unary plus
> --
>
> Key: YARN-7132
> URL: https://issues.apache.org/jira/browse/YARN-7132
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: fairscheduler
>Affects Versions: 3.0.0-alpha4
>Reporter: Daniel Templeton
>Priority: Minor
> Labels: newbie
>
> The method contains the following code:{code}
> LOG.warn(FairSchedulerConfiguration.UPDATE_INTERVAL_MS
> + " is invalid, so using default value "
> + +FairSchedulerConfiguration.DEFAULT_UPDATE_INTERVAL_MS
> + " ms instead");{code}
> Note the beginning of the third line. One of those plusses should be deleted
> so that no one else spends cycles trying to understand why it even compiles.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6877) Create an abstract log reader for extendability
[
https://issues.apache.org/jira/browse/YARN-6877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16148098#comment-16148098
]
Junping Du commented on YARN-6877:
--
Thanks Xuan for uploading a patch!
Looks like in latest patch, issue still exists for my previous comments:
{noformat}
public void testFetchApplictionLogsHar() throws Exception {
String remoteLogRootDir = "target/logs/";
-Configuration configuration = new Configuration();
+Configuration configuration = new YarnConfiguration();
{noformat}
Still lacking space between } and catch in some places:
{noformat}
}catch (NumberFormatException ne) {
{noformat}
Some new comments:
In LogAggregationWebUtils.java, verifyAndGetLogStartIndex() is better to change
to getLogStartIndex(). Normally we should check valid of input if we start with
verify*, like original methods - verifyAndGetLogLimits(), but here we just
simply throw exceptions (if input is illegal) and rely on caller to catch
exception and handle. I think call getLogStartIndex() should be better. Same
comments for verifyAndGetLogEndIndex(). But other verfiy* methods should be
fine - at least no exception for caller to catch and handle.
In LogCLIHelpers.dumpAContainerLogsForLogType():
{noformat}
+ return 0;
+} catch (IOException ex) {
+ System.err.println(ex.getMessage());
return -1;
}
{noformat}
It looks like we swallow IOException here instead of throw it out as previous
behavior. Does this expected? If not, better to keep the original way.
The same comments for dumpAllContainersLogs().
I miss one item in previous review for LogAggregationTFileController.java,
{noformat}
+ @Override
+ public void closeWriter() {
+this.writer.close();
+ }
{noformat}
Do we need to set writer here to null after we close it which is consistent
with previous behavior? The previous intention seems like to keep minimum
retain memory. The reason is still valid here. Isn't it?
Other looks fine to me.
btw, Findbugs warning shouldn't be related. Can you please check checkstyle,
javadoc warnings here?
> Create an abstract log reader for extendability
> ---
>
> Key: YARN-6877
> URL: https://issues.apache.org/jira/browse/YARN-6877
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Xuan Gong
>Assignee: Xuan Gong
> Attachments: YARN-6877-branch-2.001.patch, YARN-6877-trunk.001.patch,
> YARN-6877-trunk.002.patch, YARN-6877-trunk.003.patch,
> YARN-6877-trunk.004.patch, YARN-6877-trunk.005.patch,
> YARN-6877-trunk.006.patch
>
>
> Currently, TFile log reader is used to read aggregated log in YARN. We need
> to add an abstract layer, and pick up the correct log reader based on the
> configuration.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-7133) Clean up lock-try order fair scheduler
Daniel Templeton created YARN-7133:
--
Summary: Clean up lock-try order fair scheduler
Key: YARN-7133
URL: https://issues.apache.org/jira/browse/YARN-7133
Project: Hadoop YARN
Issue Type: Improvement
Components: fairscheduler
Affects Versions: 3.0.0-alpha4
Reporter: Daniel Templeton
There are many places that follow the pattern:{code}try {
lock.lock();
...
} finally {
lock.unlock();
}{code}
There are a couple of reasons that's a bad idea. The correct pattern
is:{code}lock.lock();
try {
...
} finally {
lock.unlock();
}{code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-7132) FairScheduler.initScheduler() contains a surprising unary plus
Daniel Templeton created YARN-7132:
--
Summary: FairScheduler.initScheduler() contains a surprising unary
plus
Key: YARN-7132
URL: https://issues.apache.org/jira/browse/YARN-7132
Project: Hadoop YARN
Issue Type: Improvement
Components: fairscheduler
Affects Versions: 3.0.0-alpha4
Reporter: Daniel Templeton
Priority: Minor
The method contains the following code:{code}
LOG.warn(FairSchedulerConfiguration.UPDATE_INTERVAL_MS
+ " is invalid, so using default value "
+ +FairSchedulerConfiguration.DEFAULT_UPDATE_INTERVAL_MS
+ " ms instead");{code}
Note the beginning of the third line. One of those plusses should be deleted
so that no one else spends cycles trying to understand why it even compiles.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6475) Fix some long function checkstyle issues
[
https://issues.apache.org/jira/browse/YARN-6475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16148060#comment-16148060
]
Miklos Szegedi commented on YARN-6475:
--
[~Soumabrata Chakraborty], could you provide the patch since you are the
original contributor?
> Fix some long function checkstyle issues
>
>
> Key: YARN-6475
> URL: https://issues.apache.org/jira/browse/YARN-6475
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Miklos Szegedi
>Assignee: Soumabrata Chakraborty
>Priority: Trivial
> Labels: newbie
> Fix For: 3.0.0-alpha4
>
> Attachments: YARN-6475.001.patch
>
>
> I am talking about these two:
> {code}
> ./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/LinuxContainerExecutor.java:441:
> @Override:3: Method length is 176 lines (max allowed is 150). [MethodLength]
> ./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java:159:
> @Override:3: Method length is 158 lines (max allowed is 150). [MethodLength]
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-6475) Fix some long function checkstyle issues
[
https://issues.apache.org/jira/browse/YARN-6475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16148060#comment-16148060
]
Miklos Szegedi edited comment on YARN-6475 at 8/30/17 9:04 PM:
---
[~soumabrata], could you provide the patch since you are the original
contributor?
was (Author: miklos.szeg...@cloudera.com):
[~Soumabrata Chakraborty], could you provide the patch since you are the
original contributor?
> Fix some long function checkstyle issues
>
>
> Key: YARN-6475
> URL: https://issues.apache.org/jira/browse/YARN-6475
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Miklos Szegedi
>Assignee: Soumabrata Chakraborty
>Priority: Trivial
> Labels: newbie
> Fix For: 3.0.0-alpha4
>
> Attachments: YARN-6475.001.patch
>
>
> I am talking about these two:
> {code}
> ./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/LinuxContainerExecutor.java:441:
> @Override:3: Method length is 176 lines (max allowed is 150). [MethodLength]
> ./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java:159:
> @Override:3: Method length is 158 lines (max allowed is 150). [MethodLength]
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6911) Graph application-level resource utilization in Web UI v2
[
https://issues.apache.org/jira/browse/YARN-6911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16148055#comment-16148055
]
Hadoop QA commented on YARN-6911:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 0m 51s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | YARN-6911 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12884541/YARN-6911.003.patch |
| Optional Tests | asflicense |
| uname | Linux 5668481b4f6d 3.13.0-119-generic #166-Ubuntu SMP Wed May 3
12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 4148023 |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/17212/console |
| Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Graph application-level resource utilization in Web UI v2
> -
>
> Key: YARN-6911
> URL: https://issues.apache.org/jira/browse/YARN-6911
> Project: Hadoop YARN
> Issue Type: New Feature
> Components: yarn-ui-v2
>Affects Versions: 3.0.0-alpha4
>Reporter: Abdullah Yousufi
>Assignee: Abdullah Yousufi
> Attachments: Resource Graph Screenshot 2.png, Resource Graph
> Screenshot.png, Resource Utilization Graph Mock Up.png, YARN-6911.001.patch,
> YARN-6911.002.patch, YARN-6911.003.patch
>
>
> It would be useful to have a visualization of the resource utilization
> (memory, cpu, etc.) per application using the ATSv2 time series data. Rough
> mock up attached.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7117) Capacity Scheduler: Support Auto Creation of Leaf Queues While Doing Queue Mapping
[
https://issues.apache.org/jira/browse/YARN-7117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16148040#comment-16148040
]
Carlo Curino commented on YARN-7117:
I have very skimmed the thread very briefly, but it appears to me that you are
looking at lots of the problems are similar to those solved by the
{{PlanQueue}} {{ReservationQueue}} and {{PlanFollower}}.
There we dealt with dynamic creation of queues, adding up to 100% as well as we
can provide (thanks to the planning aspects) stronger guarantees for the Oozie
like use cases (recurring aspects of this in YARN-5326).
I would ask you to look at that stuff, and figure out whether you can leverage
some/most of it, instead of building a parallel solution.
Overall the {{PlanFollower}} design of "observe an external signal and publish
it to scheduler" seems clean and powerful, as it decouples the CS inner
workings from outside (slower evolving) phenomena.
> Capacity Scheduler: Support Auto Creation of Leaf Queues While Doing Queue
> Mapping
> --
>
> Key: YARN-7117
> URL: https://issues.apache.org/jira/browse/YARN-7117
> Project: Hadoop YARN
> Issue Type: New Feature
> Components: capacity scheduler
>Reporter: Wangda Tan
>Assignee: Wangda Tan
>
> Currently Capacity Scheduler doesn't support auto creation of queues when
> doing queue mapping. We saw more and more use cases which has complex queue
> mapping policies configured to handle application to queues mapping.
> The most common use case of CapacityScheduler queue mapping is to create one
> queue for each user/group. However update {{capacity-scheduler.xml}} and
> {{RMAdmin:refreshQueues}} needs to be done when new user/group onboard. One
> of the option to solve the problem is automatically create queues when new
> user/group arrives.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6911) Graph application-level resource utilization in Web UI v2
[ https://issues.apache.org/jira/browse/YARN-6911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abdullah Yousufi updated YARN-6911: --- Attachment: YARN-6911.003.patch Resource Graph Screenshot 2.png Some styling changes as per [~sunilg]'s comments. I looked at the other graphs and they use squares instead of circles in their legends. Furthermore, I verified that the text in the graph is using the same font and font-size as the other graphs. > Graph application-level resource utilization in Web UI v2 > - > > Key: YARN-6911 > URL: https://issues.apache.org/jira/browse/YARN-6911 > Project: Hadoop YARN > Issue Type: New Feature > Components: yarn-ui-v2 >Affects Versions: 3.0.0-alpha4 >Reporter: Abdullah Yousufi >Assignee: Abdullah Yousufi > Attachments: Resource Graph Screenshot 2.png, Resource Graph > Screenshot.png, Resource Utilization Graph Mock Up.png, YARN-6911.001.patch, > YARN-6911.002.patch, YARN-6911.003.patch > > > It would be useful to have a visualization of the resource utilization > (memory, cpu, etc.) per application using the ATSv2 time series data. Rough > mock up attached. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-6997) org.apache.hadoop.yarn.client.SCMAdmin wrong package name
[ https://issues.apache.org/jira/browse/YARN-6997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Szegedi reassigned YARN-6997: Assignee: HondaWei > org.apache.hadoop.yarn.client.SCMAdmin wrong package name > - > > Key: YARN-6997 > URL: https://issues.apache.org/jira/browse/YARN-6997 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Reporter: Miklos Szegedi >Assignee: HondaWei >Priority: Critical > > It should be org.apache.hadoop.yarn.client.cli.SCMAdmin to follow the current > naming standard. This may cause appcompat issues in the future. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6997) org.apache.hadoop.yarn.client.SCMAdmin wrong package name
[ https://issues.apache.org/jira/browse/YARN-6997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16148016#comment-16148016 ] Miklos Szegedi commented on YARN-6997: -- Yes, please take it. > org.apache.hadoop.yarn.client.SCMAdmin wrong package name > - > > Key: YARN-6997 > URL: https://issues.apache.org/jira/browse/YARN-6997 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Reporter: Miklos Szegedi >Assignee: HondaWei >Priority: Critical > > It should be org.apache.hadoop.yarn.client.cli.SCMAdmin to follow the current > naming standard. This may cause appcompat issues in the future. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7095) Federation: routing REST invocations transparently to multiple RMs (part 3 - getNodes)
[
https://issues.apache.org/jira/browse/YARN-7095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16148001#comment-16148001
]
Subru Krishnan commented on YARN-7095:
--
Thanks [~giovanni.fumarola] for the patch. I looked at it and have a few minor
comments, all in {{FederationInterceptorREST}}:
# Use {{HadoopExecutors}} to instantiate the {{ExecutorCompletionService}} so
that we have proper logging on failures.
# Wherever you are using *executors*, please have a finally with
shutdown/cleanup.
# In *getNodes*, _deleteDuplicateNodesInfo_ is both redundant and inefficient.
Simply collect the response in a set and add to {{NodesInfo}}.
(1) & (2) are also applicable for *getApps* (YARN-7010).
> Federation: routing REST invocations transparently to multiple RMs (part 3 -
> getNodes)
> --
>
> Key: YARN-7095
> URL: https://issues.apache.org/jira/browse/YARN-7095
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Giovanni Matteo Fumarola
>Assignee: Giovanni Matteo Fumarola
> Attachments: YARN-7095.v1-rebase.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7131) FSDownload.unpack should read determine the type of resource by reading the header bytes
[ https://issues.apache.org/jira/browse/YARN-7131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brook Zhou updated YARN-7131: - Description: Currently, there are naive string checks to determine if a resource is of a particular type (jar, zip, tar.gz) There can be cases where this does not work - e.g., the user decides to split up a large zip resource as file1.zip.001, file1.zip.002. Instead, FSDownload.unpack should read the file header bytes to determine the file type. was: Currently, there are naive string checks to determine if a resource of a particular type (jar, zip, tar.gz) There can be cases where this does not work - e.g., the user decides to split up a large zip resource as file1.zip.001, file1.zip.002. Instead, FSDownload.unpack should read the file header bytes to determine the file type. > FSDownload.unpack should read determine the type of resource by reading the > header bytes > > > Key: YARN-7131 > URL: https://issues.apache.org/jira/browse/YARN-7131 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager >Reporter: Brook Zhou >Assignee: Brook Zhou > > Currently, there are naive string checks to determine if a resource is of a > particular type (jar, zip, tar.gz) > There can be cases where this does not work - e.g., the user decides to split > up a large zip resource as file1.zip.001, file1.zip.002. > Instead, FSDownload.unpack should read the file header bytes to determine the > file type. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7131) FSDownload.unpack should read determine the type of resource by reading the header bytes
[
https://issues.apache.org/jira/browse/YARN-7131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brook Zhou updated YARN-7131:
-
Description:
Currently, there are naive string checks to determine if a resource of a
particular type (jar, zip, tar.gz)
There can be cases where this does not work - e.g., the user decides to split
up a large zip resource as file1.zip.001, file1.zip.002.
Instead, FSDownload.unpack should read the file header bytes to determine the
file type.
was:
Currently, there are naive string checks to determine if a resource of a
particular type (jar, zip, tar.gz)
There can be cases where this does not work - e.g., the user decides to split
up a large zip resource as {file1}.zip.001, {file1}.zip.002.
Instead, FSDownload.unpack should read the file header bytes to determine the
file type.
> FSDownload.unpack should read determine the type of resource by reading the
> header bytes
>
>
> Key: YARN-7131
> URL: https://issues.apache.org/jira/browse/YARN-7131
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: nodemanager
>Reporter: Brook Zhou
>Assignee: Brook Zhou
>
> Currently, there are naive string checks to determine if a resource of a
> particular type (jar, zip, tar.gz)
> There can be cases where this does not work - e.g., the user decides to split
> up a large zip resource as file1.zip.001, file1.zip.002.
> Instead, FSDownload.unpack should read the file header bytes to determine the
> file type.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7131) FSDownload.unpack should read determine the type of resource by reading the header bytes
[ https://issues.apache.org/jira/browse/YARN-7131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brook Zhou updated YARN-7131: - Component/s: nodemanager > FSDownload.unpack should read determine the type of resource by reading the > header bytes > > > Key: YARN-7131 > URL: https://issues.apache.org/jira/browse/YARN-7131 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager >Reporter: Brook Zhou >Assignee: Brook Zhou > > Currently, there are naive string checks to determine if a resource is of a > particular type (jar, zip, tar.gz) > There can be cases where this does not work - e.g., the user decides to split > up a large zip resource as file1.zip.001, file1.zip.002. > Instead, FSDownload.unpack should read the file header bytes to determine the > file type. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-7131) FSDownload.unpack should read determine the type of resource by reading the header bytes
Brook Zhou created YARN-7131:
Summary: FSDownload.unpack should read determine the type of
resource by reading the header bytes
Key: YARN-7131
URL: https://issues.apache.org/jira/browse/YARN-7131
Project: Hadoop YARN
Issue Type: Improvement
Reporter: Brook Zhou
Assignee: Brook Zhou
Currently, there are naive string checks to determine if a resource of a
particular type (jar, zip, tar.gz)
There can be cases where this does not work - e.g., the user decides to split
up a large zip resource as {file1}.zip.001, {file1}.zip.002.
Instead, FSDownload.unpack should read the file header bytes to determine the
file type.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7095) Federation: routing REST invocations transparently to multiple RMs (part 3 - getNodes)
[
https://issues.apache.org/jira/browse/YARN-7095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16147983#comment-16147983
]
Hadoop QA commented on YARN-7095:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
8s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
3s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
57s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
36s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
36s{color} | {color:green} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server:
The patch generated 0 new + 17 unchanged - 2 fixed = 17 total (was 19) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 44m 50s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
3s{color} | {color:green} hadoop-yarn-server-router in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 78m 11s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestIncreaseAllocationExpirer
|
| |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation |
| | hadoop.yarn.server.resourcemanager.scheduler.fair.TestFSAppStarvation |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | YARN-7095 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12884520/YARN-7095.v1-rebase.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 1f86f60d0147 3.13.0-119-generic #166-Ubuntu SMP Wed May 3
12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / fd66a24 |
| Default Java | 1.8.0_144 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-YARN-Build/17211/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
|
| Test
[jira] [Commented] (YARN-6930) Admins should be able to explicitly enable specific LinuxContainerRuntime in the NodeManager
[ https://issues.apache.org/jira/browse/YARN-6930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16147912#comment-16147912 ] Jason Lowe commented on YARN-6930: -- Thanks for updating the patch! +1 lgtm as well. [~shaneku...@gmail.com] would you mind providing a patch for branch-2 as well? > Admins should be able to explicitly enable specific LinuxContainerRuntime in > the NodeManager > > > Key: YARN-6930 > URL: https://issues.apache.org/jira/browse/YARN-6930 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager >Reporter: Vinod Kumar Vavilapalli >Assignee: Shane Kumpf > Attachments: YARN-6930.001.patch, YARN-6930.002.patch, > YARN-6930.003.patch, YARN-6930.004.patch, YARN-6930.005.patch, > YARN-6930.006.patch > > > Today, in the java land, all LinuxContainerRuntimes are always enabled when > using LinuxContainerExecutor and the user can simply invoke anything that > he/she wants - default, docker, java-sandbox. > We should have a way for admins to explicitly enable only specific runtimes > that he/she decides for the cluster. And by default, we should have > everything other than the default one disabled. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7075) [YARN-3368] Improvement of Web UI
[ https://issues.apache.org/jira/browse/YARN-7075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16147892#comment-16147892 ] Li Lu commented on YARN-7075: - Maybe it worth the effort to make the donuts slightly thicker? If there are a lot of small pieces within one donut, the current thickness looks not enough? > [YARN-3368] Improvement of Web UI > -- > > Key: YARN-7075 > URL: https://issues.apache.org/jira/browse/YARN-7075 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Da Ding >Assignee: Da Ding > Attachments: Screen Shot 2017-08-22 at 8.36.07 PM.png, Screen Shot > 2017-08-29 at 4.36.45 PM.png, yarn-7075.001.patch > > > 1. Adjusted donut chart size to be slimmer > 2. Modified chart container style to have modern feel. > 3. Other changes like background and font. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-7095) Federation: routing REST invocations transparently to multiple RMs (part 3 - getNodes)
[
https://issues.apache.org/jira/browse/YARN-7095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16147844#comment-16147844
]
Giovanni Matteo Fumarola edited comment on YARN-7095 at 8/30/17 7:00 PM:
-
I rebased the previous patch after YARN-7010.
was (Author: giovanni.fumarola):
I rebased the previous patch after {YARN-7010}.
> Federation: routing REST invocations transparently to multiple RMs (part 3 -
> getNodes)
> --
>
> Key: YARN-7095
> URL: https://issues.apache.org/jira/browse/YARN-7095
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Giovanni Matteo Fumarola
>Assignee: Giovanni Matteo Fumarola
> Attachments: YARN-7095.v1-rebase.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7095) Federation: routing REST invocations transparently to multiple RMs (part 3 - getNodes)
[
https://issues.apache.org/jira/browse/YARN-7095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16147844#comment-16147844
]
Giovanni Matteo Fumarola commented on YARN-7095:
I rebased the previous patch after {YARN-7010}.
> Federation: routing REST invocations transparently to multiple RMs (part 3 -
> getNodes)
> --
>
> Key: YARN-7095
> URL: https://issues.apache.org/jira/browse/YARN-7095
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Giovanni Matteo Fumarola
>Assignee: Giovanni Matteo Fumarola
> Attachments: YARN-7095.v1-rebase.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7095) Federation: routing REST invocations transparently to multiple RMs (part 3 - getNodes)
[ https://issues.apache.org/jira/browse/YARN-7095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Giovanni Matteo Fumarola updated YARN-7095: --- Attachment: YARN-7095.v1-rebase.patch > Federation: routing REST invocations transparently to multiple RMs (part 3 - > getNodes) > -- > > Key: YARN-7095 > URL: https://issues.apache.org/jira/browse/YARN-7095 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Giovanni Matteo Fumarola >Assignee: Giovanni Matteo Fumarola > Attachments: YARN-7095.v1-rebase.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7095) Federation: routing REST invocations transparently to multiple RMs (part 3 - getNodes)
[ https://issues.apache.org/jira/browse/YARN-7095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Giovanni Matteo Fumarola updated YARN-7095: --- Attachment: (was: YARN-7095.v1.patch) > Federation: routing REST invocations transparently to multiple RMs (part 3 - > getNodes) > -- > > Key: YARN-7095 > URL: https://issues.apache.org/jira/browse/YARN-7095 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Giovanni Matteo Fumarola >Assignee: Giovanni Matteo Fumarola > Attachments: YARN-7095.v1-rebase.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6623) Add support to turn off launching privileged containers in the container-executor
[
https://issues.apache.org/jira/browse/YARN-6623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16147810#comment-16147810
]
Hadoop QA commented on YARN-6623:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 13 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
58s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 4m
2s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
47s{color} | {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
in trunk has 1 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
12s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
11s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
10s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} cc {color} | {color:red} 6m 10s{color} |
{color:red} hadoop-yarn-project_hadoop-yarn generated 1 new + 0 unchanged - 0
fixed = 1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
0s{color} | {color:green} hadoop-yarn-project/hadoop-yarn: The patch generated
0 new + 23 unchanged - 4 fixed = 23 total (was 27) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 4m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
54s{color} | {color:green}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
generated 0 new + 0 unchanged - 1 fixed = 0 total (was 1) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
13s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 72m 31s{color}
| {color:red} hadoop-yarn in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 13m
35s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
28s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}143m 50s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | YARN-6623 |
[jira] [Updated] (YARN-6721) container-executor should have stack checking
[ https://issues.apache.org/jira/browse/YARN-6721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated YARN-6721: --- Priority: Critical (was: Major) > container-executor should have stack checking > - > > Key: YARN-6721 > URL: https://issues.apache.org/jira/browse/YARN-6721 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager, security >Reporter: Allen Wittenauer >Assignee: Allen Wittenauer >Priority: Critical > Labels: security > Attachments: YARN-6721.00.patch > > > As per https://www.qualys.com/2017/06/19/stack-clash/stack-clash.txt and > given that container-executor is setuid, it should be compiled with stack > checking if the compiler supports such features. (-fstack-check on gcc, > -fsanitize=safe-stack on clang, -xcheck=stkovf on "Oracle Solaris Studio", > others as we find them, ...) -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6721) container-executor should have stack checking
[ https://issues.apache.org/jira/browse/YARN-6721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated YARN-6721: --- Attachment: YARN-6721.00.patch -00: * support for gcc, clang, and Sun * expects HADOOP-14670 to be applied first > container-executor should have stack checking > - > > Key: YARN-6721 > URL: https://issues.apache.org/jira/browse/YARN-6721 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager, security >Reporter: Allen Wittenauer >Assignee: Allen Wittenauer > Labels: security > Attachments: YARN-6721.00.patch > > > As per https://www.qualys.com/2017/06/19/stack-clash/stack-clash.txt and > given that container-executor is setuid, it should be compiled with stack > checking if the compiler supports such features. (-fstack-check on gcc, > -fsanitize=safe-stack on clang, -xcheck=stkovf on "Oracle Solaris Studio", > others as we find them, ...) -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-7130) ATSv2 documentation changes post merge
Varun Saxena created YARN-7130: -- Summary: ATSv2 documentation changes post merge Key: YARN-7130 URL: https://issues.apache.org/jira/browse/YARN-7130 Project: Hadoop YARN Issue Type: Bug Components: timelineserver Reporter: Varun Saxena -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6930) Admins should be able to explicitly enable specific LinuxContainerRuntime in the NodeManager
[ https://issues.apache.org/jira/browse/YARN-6930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16147748#comment-16147748 ] Eric Badger commented on YARN-6930: --- +1 (non-binding) from me on the latest patch > Admins should be able to explicitly enable specific LinuxContainerRuntime in > the NodeManager > > > Key: YARN-6930 > URL: https://issues.apache.org/jira/browse/YARN-6930 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager >Reporter: Vinod Kumar Vavilapalli >Assignee: Shane Kumpf > Attachments: YARN-6930.001.patch, YARN-6930.002.patch, > YARN-6930.003.patch, YARN-6930.004.patch, YARN-6930.005.patch, > YARN-6930.006.patch > > > Today, in the java land, all LinuxContainerRuntimes are always enabled when > using LinuxContainerExecutor and the user can simply invoke anything that > he/she wants - default, docker, java-sandbox. > We should have a way for admins to explicitly enable only specific runtimes > that he/she decides for the cluster. And by default, we should have > everything other than the default one disabled. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-5355) YARN Timeline Service v.2: alpha 2
[
https://issues.apache.org/jira/browse/YARN-5355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16147723#comment-16147723
]
Vrushali C edited comment on YARN-5355 at 8/30/17 6:16 PM:
---
The merge vote concluded as noted at
http://www.mail-archive.com/yarn-dev@hadoop.apache.org/msg27610.html
and accordingly Timeline Service v2 has been merged to trunk as part of
https://github.com/apache/hadoop/commit/32cba6c3036dfb1dcb0a4fd77a68ddad17dd4082
Three cheers for [~rohithsharma], [~varun_saxena], [~sjlee0], [~haibo.chen],
[~jrottinghuis], [~gtCarrera9] and [~vrushalic] for contributing to this
milestone!
Special thanks to folks who guided us with their insights, reviews and
discussions:
[~jianhe], [~vinodkv], [~jlowe], [~rkanter], [~djp], [~stack], [~enis],
[~subru], [~curino], [~ctrezzo], [~lohit] and to [~andrew.wang] for including
it as part of 3.0.0-beta1.
For further updates on progress on Timeline Service v2, please follow YARN-7055
was (Author: vrushalic):
The merge vote concluded as noted at
http://www.mail-archive.com/yarn-dev@hadoop.apache.org/msg27610.html
and accordingly Timeline Service v2 has been merged to trunk as part of
https://github.com/apache/hadoop/commit/32cba6c3036dfb1dcb0a4fd77a68ddad17dd4082
Three cheers for [~rohithsharma] [~varun_saxena] [~sjlee0] [~haibo.chen]
[~jrottinghuis] [~gtCarrera9] and [~vrushalic] for contributing to this
milestone!
Special thanks to folks who guided us with their insights, reviews and
discussions:
[~jianhe] [~vinodkv] [~jlowe] [~rkanter] [~djp] [~stack] [~enis] [~subru]
[~curino] [~ctrezzo] [~lohit] and to [~andrew.wang] for including it as part of
3.0.0-beta1.
For further updates on progress on Timeline Service v2, please follow YARN-7055
> YARN Timeline Service v.2: alpha 2
> --
>
> Key: YARN-5355
> URL: https://issues.apache.org/jira/browse/YARN-5355
> Project: Hadoop YARN
> Issue Type: New Feature
> Components: timelineserver
>Reporter: Sangjin Lee
>Assignee: Vrushali C
>Priority: Critical
> Fix For: 3.0.0-beta1
>
> Attachments: Documentation - The YARN Timeline Service v2.pdf,
> Timeline Service v2_ Ideas for Next Steps.pdf, YARN-5355.01.patch,
> YARN-5355.02.patch, YARN-5355.03.patch, YARN-5355-branch-2.01.patch
>
>
> This is an umbrella JIRA for the alpha 2 milestone for YARN Timeline Service
> v.2.
> This is developed on feature branches: {{YARN-5355}} for the trunk-based
> development and {{YARN-5355-branch-2}} to maintain backports to branch-2. Any
> subtask work on this JIRA will be committed to those 2 branches.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7094) Document that server-side graceful decom is currently not recommended
[
https://issues.apache.org/jira/browse/YARN-7094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16147719#comment-16147719
]
Hadoop QA commented on YARN-7094:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 5m
22s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
48s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
7s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m
35s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
58s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
52s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
39s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 20m
49s{color} | {color:green} hadoop-yarn-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
16s{color} | {color:green} hadoop-yarn-site in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
30s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 72m 18s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | YARN-7094 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12884383/YARN-7094.002.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 4b881ae61f3a 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31
14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / a20e710 |
| Default Java | 1.8.0_144 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/17210/testReport/ |
|
[jira] [Commented] (YARN-7073) Rest API site documentation
[
https://issues.apache.org/jira/browse/YARN-7073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16147714#comment-16147714
]
Gour Saha commented on YARN-7073:
-
[~jianhe], the 003 patch looks good. Few comments -
1.
{code}
+ summary: Get a service details
{code}
change to -
{code}
+ summary: Get details of a service
{code}
2.
In addition to lifetime update, PUT also supports flex up/down and stop/start
of a service. There is an example of flex up/down but is not captured in the
swagger definition. Also, neither the swagger definition nor the examples
mention that PUT supports stop/start of a service. I think we need to update
the text _*"As of now, only update of lifetime is supported."*_ as shown below
and mention about everything that PUT supports. The mention about upgrade
operation is marked as TBD already, so we should leave it as is, since our
users should know that it is an important feature and we will implement it soon.
{code}
/services/{service_name}:
+put:
+ summary: Update a service or upgrade the binary version of the
components of a running service
+ description: Update the runtime properties of a service. As of now, only
update of lifetime is supported. The PUT operation is also used to orchestrate
an upgrade of the service containers to a newer version of their artifacts(TBD)
.
{code}
change to -
{code}
/services/{service_name}:
+put:
+ summary: Update a service or upgrade the binary version of the
components of a running service
+ description: Update the runtime properties of a service. Currently the
following operations are supported - update lifetime, flex up/down of one or
more components, and stop/start a service. The PUT operation is also used to
orchestrate an upgrade of the service containers to a newer version of their
artifacts (TBD).
{code}
Examples of stop and start using PUT are as below -
{code}
### Stop a service
PUT URL - http://host.mycompany.com:9191/ws/v1/services/hello-world
# PUT Request JSON
```json
{
"state": "STOPPED"
}
```
{code}
{code}
### Start a service
PUT URL - http://host.mycompany.com:9191/ws/v1/services/hello-world
# PUT Request JSON
```json
{
"state": "STARTED",
"lifetime": 3600
}
```
{code}
3.
The mention of _*"Not implemeted yet."*_ is a little ambiguous which can make
users think that we don't support delete. So slight change there.
{code}
+delete:
+ summary: Destroy a service
+ description: Destroy a service and release all resources. This API might
have to return JSON data providing location of logs, etc. Not implemeted yet.
{code}
change to -
{code}
+delete:
+ summary: Destroy a service
+ description: Destroy a service and release all resources. This API might
have to return JSON data providing location of logs, etc (TBD).
{code}
> Rest API site documentation
> ---
>
> Key: YARN-7073
> URL: https://issues.apache.org/jira/browse/YARN-7073
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: documentation, site
>Reporter: Gour Saha
>Assignee: Gour Saha
> Attachments: YARN-7073-yarn-native-services.001.patch,
> YARN-7073-yarn-native-services.002.patch,
> YARN-7073-yarn-native-services.003.patch
>
>
> Commit site documentation for REST API service, generated from the swagger
> definition as a MD file.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6623) Add support to turn off launching privileged containers in the container-executor
[
https://issues.apache.org/jira/browse/YARN-6623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16147713#comment-16147713
]
Eric Badger commented on YARN-6623:
---
Hey [~vvasudev], thanks for the update. It looks like most of my first round
comments were lost, so posting them again.
{noformat}
int is_docker_support_enabled() {
-return is_feature_enabled(DOCKER_SUPPORT_ENABLED_KEY,
-DEFAULT_DOCKER_SUPPORT_ENABLED, _cfg);
+ return is_feature_enabled(DOCKER_SUPPORT_ENABLED_KEY,
+DEFAULT_DOCKER_SUPPORT_ENABLED, _cfg)
+ || docker_module_enabled();
{noformat}
The indentation here should be fixed.
{noformat}
+# The configs below deal with settings for Docker
+#[docker]
+# module.enabled=## enable/disable the module. set to "true" to enable,
disabled by default
+# docker.binary=/usr/bin/docker
+# allowed.capabilities=## comma seperated capabilities that can be granted,
e.g
CHOWN,DAC_OVERRIDE,FSETID,FOWNER,MKNOD,NET_RAW,SETGID,SETUID,SETFCAP,SETPCAP,NET_BIND_SERVICE,SYS_CHROOT,KILL,AUDIT_WRITE
+# allowed.devices=## comma seperated list of devices that can be mounted into
a container
+# allowed.networks=## comma seperated networks that can be used. e.g
bridge,host,none
+# allowed.ro-mounts=## comma seperated volumes that can be mounted as
read-only
+# allowed.rw-mounts=## comma seperate volumes that can be mounted as
read-write, add the yarn local and log dirs to this list to run Hadoop jobs
{noformat}
For the Docker configs, I think it'd be more clear if they were prefixed with
docker.
{noformat}
+
+
+
+
+
+
{noformat}
Since we're going to remove the hard-coded string in YARN-6968, should we leave
the findbugs warning and take care of it there?
{noformat}
+ protected final void addCommandArguments(String key, String value) {
+if (commandArguments.containsKey(key)) {
+ commandArguments.get(key).add(value);
+ return;
+}
{noformat}
No need to call contains() and then call get(). We can just call get and then
add the value if the return value isn't null.
{noformat}
+ @Override
+ public String toString() {
+StringBuffer ret = new StringBuffer("");
+ret.append(this.command);
{noformat}
Any reason not to create the string with the command in the constructor?
Something like
{{StringBuffer ret = new StringBuffer(this.command);}}
> Add support to turn off launching privileged containers in the
> container-executor
> -
>
> Key: YARN-6623
> URL: https://issues.apache.org/jira/browse/YARN-6623
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager
>Reporter: Varun Vasudev
>Assignee: Varun Vasudev
>Priority: Blocker
> Attachments: YARN-6623.001.patch, YARN-6623.002.patch,
> YARN-6623.003.patch, YARN-6623.004.patch, YARN-6623.005.patch,
> YARN-6623.006.patch, YARN-6623.007.patch
>
>
> Currently, launching privileged containers is controlled by the NM. We should
> add a flag to the container-executor.cfg allowing admins to disable launching
> privileged containers at the container-executor level.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-2497) Changes for fair scheduler to support allocate resource respect labels
[ https://issues.apache.org/jira/browse/YARN-2497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16147663#comment-16147663 ] Daniel Templeton edited comment on YARN-2497 at 8/30/17 5:41 PM: - bq. We should guarantee that a queue with non-label cannot access a node with a label Agreed, and the current patch does that. What I still have to figure out is how to sensibly assign a queue labels and no label. In capacity scheduler, all queues can access nodes with no label. I'm not sure that's the best approach. For example, assume I have a GPU label, and I want to make sure that any app requesting nodes with the GPU label is scheduled as a priority (because my GPU card is an expensive resource that I want to see maximally used). I therefore create a GPU queue and give that queue a very high weight. If that queue also allowed apps with no label, then I could submit non-GPU jobs to that queue just to boost my priority. On the other hand, I want to be able to submit an app that uses no label for the AM so that I don't consume GPU resources for no reason. I still need to ponder that one a little. Multiple labels are explicitly not supported because of the chaos that would create. Instead see YARN-3409. I do not intend to tackle relaxed partitions in this patch. That's a much trickier implementation that requires delayed scheduling. Feel free to file a JIRA for it and work on it. I will be testing failover for node labels, but I don't see any reason why it shouldn't work as is. bq. A queue should only access one label That doesn't work. Because an app can only be in one queue at a time, in order for an app to use different labels for different containers, the queue must support multiple labels. A primary use case is as I mentioned above, an AM that doesn't want to consume a limited resource that its tasks will need. I don't like it either, but I don't see another way around it. was (Author: templedf): bq. We should guarantee that a queue with non-label cannot access a node with a label Agreed, and the current patch does that. What I still have to figure out is how to sensibly assign a queue labels and no label. In capacity scheduler, all queues can access nodes with no label. I'm not sure that's the best approach. For example, assume I have a GPU label, and I want to make sure that any app requesting nodes with the GPU label is scheduled as a priority (because my GPU card is an expensive resource that I want to see maximally used). I therefore create a GPU queue and give that queue a very high weight. If that queue also allowed apps with no label, then I could submit non-GPU jobs to that queue just to boost my priority. On the other hand, I want to be able to submit an app that uses no label for the AM so that I don't consume GPU resources for no reason. I still need to ponder that one a little. Multiple labels are explicitly not supported because of the chaos that would create. Instead see YARN-3409. I do not intend to tackle relaxed partitions in this patch. That's a much trickier implementation that requires delayed scheduling. I will be testing failover for node labels, but I don't see any reason why it shouldn't work as is. bq. A queue should only access one label That doesn't work. Because an app can only be in one queue at a time, in order for an app to use different labels for different containers, the queue must support multiple labels. A primary use case is as I mentioned above, an AM that doesn't want to consume a limit resource that its tasks will need. I don't like it either, but I don't see another way around it. > Changes for fair scheduler to support allocate resource respect labels > -- > > Key: YARN-2497 > URL: https://issues.apache.org/jira/browse/YARN-2497 > Project: Hadoop YARN > Issue Type: Sub-task > Components: fairscheduler >Reporter: Wangda Tan >Assignee: Daniel Templeton > Attachments: YARN-2499.WIP01.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-2497) Changes for fair scheduler to support allocate resource respect labels
[ https://issues.apache.org/jira/browse/YARN-2497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16147663#comment-16147663 ] Daniel Templeton commented on YARN-2497: bq. We should guarantee that a queue with non-label cannot access a node with a label Agreed, and the current patch does that. What I still have to figure out is how to sensibly assign a queue labels and no label. In capacity scheduler, all queues can access nodes with no label. I'm not sure that's the best approach. For example, assume I have a GPU label, and I want to make sure that any app requesting nodes with the GPU label is scheduled as a priority (because my GPU card is an expensive resource that I want to see maximally used). I therefore create a GPU queue and give that queue a very high weight. If that queue also allowed apps with no label, then I could submit non-GPU jobs to that queue just to boost my priority. On the other hand, I want to be able to submit an app that uses no label for the AM so that I don't consume GPU resources for no reason. I still need to ponder that one a little. Multiple labels are explicitly not supported because of the chaos that would create. Instead see YARN-3409. I do not intend to tackle relaxed partitions in this patch. That's a much trickier implementation that requires delayed scheduling. I will be testing failover for node labels, but I don't see any reason why it shouldn't work as is. bq. A queue should only access one label That doesn't work. Because an app can only be in one queue at a time, in order for an app to use different labels for different containers, the queue must support multiple labels. A primary use case is as I mentioned above, an AM that doesn't want to consume a limit resource that its tasks will need. I don't like it either, but I don't see another way around it. > Changes for fair scheduler to support allocate resource respect labels > -- > > Key: YARN-2497 > URL: https://issues.apache.org/jira/browse/YARN-2497 > Project: Hadoop YARN > Issue Type: Sub-task > Components: fairscheduler >Reporter: Wangda Tan >Assignee: Daniel Templeton > Attachments: YARN-2499.WIP01.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6830) Support quoted strings for environment variables
[
https://issues.apache.org/jira/browse/YARN-6830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16147651#comment-16147651
]
Daniel Templeton commented on YARN-6830:
How about this regex: {code}"^(" + Shell.ENV_NAME_REGEX +
")=((['\"])(((?!\\3).)*)\\3|[^,])(,|$)"{code}. In a regex tester it works.
> Support quoted strings for environment variables
>
>
> Key: YARN-6830
> URL: https://issues.apache.org/jira/browse/YARN-6830
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Shane Kumpf
>Assignee: Shane Kumpf
> Attachments: YARN-6830.001.patch
>
>
> There are cases where it is necessary to allow for quoted string literals
> within environment variables values when passed via the yarn command line
> interface.
> For example, consider the follow environment variables for a MR map task.
> {{MODE=bar}}
> {{IMAGE_NAME=foo}}
> {{MOUNTS=/tmp/foo,/tmp/bar}}
> When running the MR job, these environment variables are supplied as a comma
> delimited string.
> {{-Dmapreduce.map.env="MODE=bar,IMAGE_NAME=foo,MOUNTS=/tmp/foo,/tmp/bar"}}
> In this case, {{MOUNTS}} will be parsed and added to the task environment as
> {{MOUNTS=/tmp/foo}}. Any attempts to quote the embedded comma separated value
> results in quote characters becoming part of the value, and parsing still
> breaks down at the comma.
> This issue is to allow for quoting the comma separated value (escaped double
> or single quote). This was mentioned on YARN-4595 and will impact YARN-5534
> as well.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6933) ResourceUtils.DISALLOWED_NAMES and ResourceUtils.checkMandatoryResources() are duplicating work
[
https://issues.apache.org/jira/browse/YARN-6933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16147646#comment-16147646
]
Hadoop QA commented on YARN-6933:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
21s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} YARN-3926 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
45s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
28s{color} | {color:green} YARN-3926 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m
11s{color} | {color:green} YARN-3926 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
2s{color} | {color:green} YARN-3926 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
22s{color} | {color:green} YARN-3926 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
29s{color} | {color:green} YARN-3926 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
9s{color} | {color:green} YARN-3926 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
34s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
32s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
28s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 58m 9s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | YARN-6933 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12884504/YARN-6933-YARN-3926.009.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle xml |
| uname | Linux 4ecef5628fbb 3.13.0-119-generic #166-Ubuntu SMP Wed May 3
12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | YARN-3926 / 38d04b8 |
| Default Java | 1.8.0_144 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/17209/testReport/ |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common U:
hadoop-yarn-project/hadoop-yarn |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/17209/console |
| Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org |
This message was automatically
[jira] [Updated] (YARN-7129) Application Catalog for YARN applications
[ https://issues.apache.org/jira/browse/YARN-7129?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Yang updated YARN-7129: Attachment: YARN Appstore.pdf Design document for YARN Appstore > Application Catalog for YARN applications > - > > Key: YARN-7129 > URL: https://issues.apache.org/jira/browse/YARN-7129 > Project: Hadoop YARN > Issue Type: New Feature > Components: applications >Affects Versions: 3.0.0-beta1 >Reporter: Eric Yang >Assignee: Eric Yang > Attachments: YARN Appstore.pdf > > > YARN native services provides web services API to improve usability of > application deployment on Hadoop using collection of docker images. It would > be nice to have an application catalog system which provides an editorial and > search interface for YARN applications. This improves usability of YARN for > manage the life cycle of applications. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6830) Support quoted strings for environment variables
[
https://issues.apache.org/jira/browse/YARN-6830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16147633#comment-16147633
]
Shane Kumpf commented on YARN-6830:
---
No problem. Thanks for taking another look, [~templedf]. I've tried what you've
provided as-is and with a couple modifications and again it is close, but it
breaks down on the non-quoted values. Let me know if you have additional
suggestions to simplify this into a single regex and i'd be happy to try them
out.
> Support quoted strings for environment variables
>
>
> Key: YARN-6830
> URL: https://issues.apache.org/jira/browse/YARN-6830
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Shane Kumpf
>Assignee: Shane Kumpf
> Attachments: YARN-6830.001.patch
>
>
> There are cases where it is necessary to allow for quoted string literals
> within environment variables values when passed via the yarn command line
> interface.
> For example, consider the follow environment variables for a MR map task.
> {{MODE=bar}}
> {{IMAGE_NAME=foo}}
> {{MOUNTS=/tmp/foo,/tmp/bar}}
> When running the MR job, these environment variables are supplied as a comma
> delimited string.
> {{-Dmapreduce.map.env="MODE=bar,IMAGE_NAME=foo,MOUNTS=/tmp/foo,/tmp/bar"}}
> In this case, {{MOUNTS}} will be parsed and added to the task environment as
> {{MOUNTS=/tmp/foo}}. Any attempts to quote the embedded comma separated value
> results in quote characters becoming part of the value, and parsing still
> breaks down at the comma.
> This issue is to allow for quoting the comma separated value (escaped double
> or single quote). This was mentioned on YARN-4595 and will impact YARN-5534
> as well.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6953) Clean up ResourceUtils.setMinimumAllocationForMandatoryResources() and setMaximumAllocationForMandatoryResources()
[
https://issues.apache.org/jira/browse/YARN-6953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16147614#comment-16147614
]
Daniel Templeton commented on YARN-6953:
Getting better. Still seems rather complex. To add to the confusion, here's
what I would do:
{code}
private static void setAllocationForMandatoryResources(
Map res, Configuration conf) {
ResourceInformation mem = res.get(ResourceInformation.MEMORY_MB.getName());
if (mem == null) {
//complain
} else {
mem.setMinimumAllocation(getAllocation(conf,
YarnConfiguration.RESOURCE_TYPES + "." + mem.getName() +
MINIMUM_ALLOCATION,
YarnConfiguration.RM_SCHEDULER_MINIMUM_ALLOCATION_MB,
YarnConfiguration.DEFAULT_RM_SCHEDULER_MINIMUM_ALLOCATION_MB));
mem.setMaximumAllocation(getAllocation(conf,
YarnConfiguration.RESOURCE_TYPES + "." + mem.getName() +
MAXIMUM_ALLOCATION,
YarnConfiguration.RM_SCHEDULER_MAXIMUM_ALLOCATION_MB,
YarnConfiguration.DEFAULT_RM_SCHEDULER_MAXIMUM_ALLOCATION_MB));
}
ResourceInformation cpu = res.get(ResourceInformation.MEMORY_MB.getName());
if (cpu == null) {
//complain
} else {
cpu.setMinimumAllocation(getAllocation(conf,
YarnConfiguration.RESOURCE_TYPES + "." + cpu.getName() +
MINIMUM_ALLOCATION,
YarnConfiguration.RM_SCHEDULER_MINIMUM_ALLOCATION_VCORES,
YarnConfiguration.DEFAULT_RM_SCHEDULER_MINIMUM_ALLOCATION_VCORES));
cpu.setMaximumAllocation(getAllocation(conf,
YarnConfiguration.RESOURCE_TYPES + "." + cpu.getName() +
MAXIMUM_ALLOCATION,
YarnConfiguration.RM_SCHEDULER_MAXIMUM_ALLOCATION_VCORES,
YarnConfiguration.DEFAULT_RM_SCHEDULER_MAXIMUM_ALLOCATION_VCORES));
}
}
private static void getAllocation(Configuration conf, String resourceTypesKey,
String schedulerKey, long schedulerDefault) {
long value = conf.getLong(resourceTypesKey, -1L);
if (value == -1) {
value = conf.getLong(schedulerKey, schedulerDefault);
}
return value;
}
{code}
I didn't actually look to see if the null checks are needed.
This code is a little longer, but for me it's clearer. I don't have to do any
mental gymnastics to understand what it's doing.
> Clean up ResourceUtils.setMinimumAllocationForMandatoryResources() and
> setMaximumAllocationForMandatoryResources()
> --
>
> Key: YARN-6953
> URL: https://issues.apache.org/jira/browse/YARN-6953
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Affects Versions: YARN-3926
>Reporter: Daniel Templeton
>Assignee: Manikandan R
>Priority: Minor
> Labels: newbie
> Attachments: YARN-6953-YARN-3926.001.patch,
> YARN-6953-YARN-3926.002.patch, YARN-6953-YARN-3926.003.patch,
> YARN-6953-YARN-3926.004.patch, YARN-6953-YARN-3926-WIP.patch
>
>
> The {{setMinimumAllocationForMandatoryResources()}} and
> {{setMaximumAllocationForMandatoryResources()}} methods are quite convoluted.
> They'd be much simpler if they just handled CPU and memory manually instead
> of trying to be clever about doing it in a loop. There are also issues, such
> as the log warning always talking about memory or the last element of the
> inner array being a copy of the first element.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6933) ResourceUtils.DISALLOWED_NAMES and ResourceUtils.checkMandatoryResources() are duplicating work
[
https://issues.apache.org/jira/browse/YARN-6933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16147579#comment-16147579
]
Daniel Templeton commented on YARN-6933:
Fine by me. I'll let [~sunilg] have the final say.
> ResourceUtils.DISALLOWED_NAMES and ResourceUtils.checkMandatoryResources()
> are duplicating work
> ---
>
> Key: YARN-6933
> URL: https://issues.apache.org/jira/browse/YARN-6933
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Affects Versions: YARN-3926
>Reporter: Daniel Templeton
>Assignee: Manikandan R
> Labels: newbie++
> Attachments: YARN-6933-YARN-3926.001.patch,
> YARN-6933-YARN-3926.002.patch, YARN-6933-YARN-3926.003.patch,
> YARN-6933-YARN-3926.004.patch, YARN-6933-YARN-3926.005.patch,
> YARN-6933-YARN-3926.006.patch, YARN-6933-YARN-3926.007.patch,
> YARN-6933-YARN-3926.008.patch, YARN-6933-YARN-3926.009.patch
>
>
> Both are used to check that the mandatory resources were not redefined. Only
> one check is needed. I would recommend dropping {{DISALLOWED_RESOURCES}}.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6868) Add test scope to certain entries in hadoop-yarn-server-resourcemanager pom.xml
[
https://issues.apache.org/jira/browse/YARN-6868?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16147575#comment-16147575
]
Hudson commented on YARN-6868:
--
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #12274 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/12274/])
YARN-6868. Add test scope to certain entries in (haibochen: rev
a20e7105ea9d4e38d7f8f9fd48035e342bb22f1c)
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/pom.xml
> Add test scope to certain entries in hadoop-yarn-server-resourcemanager
> pom.xml
> ---
>
> Key: YARN-6868
> URL: https://issues.apache.org/jira/browse/YARN-6868
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
>Affects Versions: 3.0.0-beta1
>Reporter: Ray Chiang
>Assignee: Ray Chiang
> Fix For: 3.0.0-beta1
>
> Attachments: YARN-6868.001.patch
>
>
> The tag
> {noformat}
> test
> {noformat}
> is missing from a few entries in the pom.xml for
> hadoop-yarn-server-resourcemanager.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7123) FairScheduler.getResourceCalculator() returns an instance of DefaultResourceCalculator regardless of the configuration
[
https://issues.apache.org/jira/browse/YARN-7123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16147573#comment-16147573
]
Daniel Templeton commented on YARN-7123:
Looks generally good. Before we go fixing the issue, that, I'd like some
independent verification that the issue is really an issue. On the patch,
please add a default to the docs for the new property. Also, it would be
better to add a default class to {{FairSchedulerConfiguration}} than to add the
dependency on {{CapacitySchedulerConfiguration}}.
> FairScheduler.getResourceCalculator() returns an instance of
> DefaultResourceCalculator regardless of the configuration
> --
>
> Key: YARN-7123
> URL: https://issues.apache.org/jira/browse/YARN-7123
> Project: Hadoop YARN
> Issue Type: Bug
> Components: fairscheduler
>Affects Versions: 3.0.0-alpha4
>Reporter: Daniel Templeton
> Attachments: YARN-7123.branch-3.0.0-alpha4.001.patch
>
>
> There are several places where this creates the wrong behavior:
> * 298:RMServerUtils.java
> * 1081:AbstractYarnScheduler.java
> * 1197:FSAppAttempt.java
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
