[jira] [Commented] (YARN-9483) DistributedShell does not release container when failed to localize at launch
[
https://issues.apache.org/jira/browse/YARN-9483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16819777#comment-16819777
]
Prabhu Joseph commented on YARN-9483:
-
[~pbacsko] Can you review this jira when you get time. Thanks.
> DistributedShell does not release container when failed to localize at launch
> -
>
> Key: YARN-9483
> URL: https://issues.apache.org/jira/browse/YARN-9483
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Major
> Attachments: YARN-9483-001.patch
>
>
> DistributedShell does not release container when failed to localize at
> launch. The launch threads does not increment completed & failed containers
> when failed to localize. And the main thread waits for the containers to
> complete without failing the job.
> {code}
> yarn jar
> /HADOOP/hadoop-3.2.0/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-3.2.0.jar
> -shell_command ls -shell_args / -jar
> /HADOOP/hadoop-3.2.0/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-3.2.0.jar
> -localize_files /tmp/prabhu
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9472) Add multi-thread asynchronous scheduling to fair scheduler
[ https://issues.apache.org/jira/browse/YARN-9472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16819765#comment-16819765 ] zhuqi commented on YARN-9472: - Hi, [~Tao Yang] . Thanks for your reply. Yes, i mean the final multi-thread asynchronous scheduling can be supported in global scheduler base request-based. And the non-HB driven approach that continuous scheduling has be removed because of the lock contention in large scale (YARN-6486), the development of global scheduling for fair scheduler should be a long time . If we can first add a RM service for scheduling performance improvements only, but not consider all the complex resource placement requirements, which can be coupled with some simple localilty relaxation strategy? > Add multi-thread asynchronous scheduling to fair scheduler > -- > > Key: YARN-9472 > URL: https://issues.apache.org/jira/browse/YARN-9472 > Project: Hadoop YARN > Issue Type: Sub-task > Components: fairscheduler, resourcemanager >Reporter: zhuqi >Priority: Major > > Now the capacity scheduler has multi-thread asynchronous scheduling, i think > the fair scheduler also need to support it. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-9306) Detect docker image existence during container launch
[ https://issues.apache.org/jira/browse/YARN-9306?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bilwa S T reassigned YARN-9306: --- Assignee: Bilwa S T > Detect docker image existence during container launch > - > > Key: YARN-9306 > URL: https://issues.apache.org/jira/browse/YARN-9306 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Eric Yang >Assignee: Bilwa S T >Priority: Major > > It would be good to check yarn.nodemanager.runtime.linux.docker.image-update > flag. When the flag is false, and docker image doesn't exist in docker > cache. Container launch should abort, and try on another node. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9440) Improve diagnostics for scheduler and app activities
[
https://issues.apache.org/jira/browse/YARN-9440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16819730#comment-16819730
]
Hadoop QA commented on YARN-9440:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
14s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 7 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
49s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
34s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
57s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
16s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 38s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
22s{color} | {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
in trunk has 2 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
34s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
14s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m
49s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 10s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch
generated 5 new + 196 unchanged - 19 fixed = 201 total (was 215) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 53s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
30s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
54s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 80m
46s{color} | {color:green} hadoop-yarn-server-resourcemanager in the patch
passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
40s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}162m 28s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:bdbca0e |
| JIRA Issue | YARN-9440 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12966175/YARN-9440.002.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 162e4e227a6f 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24
10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 04c0437 |
| maven | version: Apache
[jira] [Commented] (YARN-9490) applicationresourceusagereport return wrong number of reserved containers
[ https://issues.apache.org/jira/browse/YARN-9490?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16819703#comment-16819703 ] Tao Yang commented on YARN-9490: Thanks [~zyb] for raising this issue. The patch makes sense to me, just a small suggestion: we can iterate values if no need to access the keys, and a simpler way is to use stream like this: this.reservedContainers.values().stream().mapToInt(Map::size).sum(); > applicationresourceusagereport return wrong number of reserved containers > - > > Key: YARN-9490 > URL: https://issues.apache.org/jira/browse/YARN-9490 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 3.3.0 >Reporter: yanbing zhang >Priority: Minor > Fix For: 3.3.0 > > Attachments: YARN-9490.patch > > > when getting an ApplicationResourceUsageReport instance from the class of > SchedulerApplicationAttempt, I found the input constructor > parameter(reservedContainers.size()) is wrong. because the type of this > variable is Map>, so > "reservedContainer.size()" is not the number of containers, but the number of > SchedulerRequestKey. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9489) Support filtering by request-priorities and allocation-request-ids for query results of app activities
[ https://issues.apache.org/jira/browse/YARN-9489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Yang updated YARN-9489: --- Attachment: YARN-9489.001.patch > Support filtering by request-priorities and allocation-request-ids for query > results of app activities > -- > > Key: YARN-9489 > URL: https://issues.apache.org/jira/browse/YARN-9489 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Tao Yang >Assignee: Tao Yang >Priority: Major > Attachments: YARN-9489.001.patch > > > [Design Doc > #4.2|https://docs.google.com/document/d/1pwf-n3BCLW76bGrmNPM4T6pQ3vC4dVMcN2Ud1hq1t2M/edit#heading=h.m04tqsosk94h] -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9440) Improve diagnostics for scheduler and app activities
[ https://issues.apache.org/jira/browse/YARN-9440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Yang updated YARN-9440: --- Description: [Design doc #4.1|https://docs.google.com/document/d/1pwf-n3BCLW76bGrmNPM4T6pQ3vC4dVMcN2Ud1hq1t2M/edit#heading=h.cyw6zeehzqmx] (was: [Design doc|https://docs.google.com/document/d/1pwf-n3BCLW76bGrmNPM4T6pQ3vC4dVMcN2Ud1hq1t2M/edit#heading=h.cyw6zeehzqmx] ) > Improve diagnostics for scheduler and app activities > > > Key: YARN-9440 > URL: https://issues.apache.org/jira/browse/YARN-9440 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacityscheduler >Reporter: Tao Yang >Assignee: Tao Yang >Priority: Major > Attachments: YARN-9440.001.patch, YARN-9440.002.patch > > > [Design doc > #4.1|https://docs.google.com/document/d/1pwf-n3BCLW76bGrmNPM4T6pQ3vC4dVMcN2Ud1hq1t2M/edit#heading=h.cyw6zeehzqmx] > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9440) Improve diagnostics for scheduler and app activities
[ https://issues.apache.org/jira/browse/YARN-9440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16819674#comment-16819674 ] Tao Yang commented on YARN-9440: Thanks [~cheersyang]. Attached v2 patch to fix UT failures which are caused by the modification of diagnostics (a separator in diagnostics have changed from ":" to "="). > Improve diagnostics for scheduler and app activities > > > Key: YARN-9440 > URL: https://issues.apache.org/jira/browse/YARN-9440 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacityscheduler >Reporter: Tao Yang >Assignee: Tao Yang >Priority: Major > Attachments: YARN-9440.001.patch, YARN-9440.002.patch > > > [Design > doc|https://docs.google.com/document/d/1pwf-n3BCLW76bGrmNPM4T6pQ3vC4dVMcN2Ud1hq1t2M/edit#heading=h.cyw6zeehzqmx] > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9440) Improve diagnostics for scheduler and app activities
[ https://issues.apache.org/jira/browse/YARN-9440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Yang updated YARN-9440: --- Attachment: YARN-9440.002.patch > Improve diagnostics for scheduler and app activities > > > Key: YARN-9440 > URL: https://issues.apache.org/jira/browse/YARN-9440 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacityscheduler >Reporter: Tao Yang >Assignee: Tao Yang >Priority: Major > Attachments: YARN-9440.001.patch, YARN-9440.002.patch > > > [Design > doc|https://docs.google.com/document/d/1pwf-n3BCLW76bGrmNPM4T6pQ3vC4dVMcN2Ud1hq1t2M/edit#heading=h.cyw6zeehzqmx] > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9349) When doTransition() method occurs exception, the log level practices are inconsistent
[
https://issues.apache.org/jira/browse/YARN-9349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16819614#comment-16819614
]
Hudson commented on YARN-9349:
--
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #16425 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/16425/])
YARN-9349. Improved log level practices for (eyang: rev
9cf7401794def0d420876db5db20fcd76c52193f)
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/application/ApplicationImpl.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/LocalizedResource.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/container/ContainerImpl.java
> When doTransition() method occurs exception, the log level practices are
> inconsistent
> -
>
> Key: YARN-9349
> URL: https://issues.apache.org/jira/browse/YARN-9349
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: nodemanager
>Affects Versions: 3.1.0, 2.8.5
>Reporter: Anuhan Torgonshar
>Priority: Major
> Labels: easyfix
> Fix For: 3.3.0
>
> Attachments: YARN-9349.trunk.patch
>
>
> There are *inconsistent* log level practices when code catches
> *_InvalidStateTransitionException_* for _*doTransition()*_ method.
> {code:java}
> **WARN level**
> /**
> file path:
> hadoop-2.8.5-src\hadoop-yarn-project\hadoop-yarn\hadoop-yarn-server\hadoop-yarn-server-nodemanager\src\main\java\org\apache\hadoop\yarn\server\nodemanager\containermanager\application\ApplicationImpl.java
> log statement line number: 482
> log level:warn
> **/
> try {
>// queue event requesting init of the same app
>newState = stateMachine.doTransition(event.getType(), event);
> } catch (InvalidStateTransitionException e) {
>LOG.warn("Can't handle this event at current state", e);
> }
> /**
> file path:
> hadoop-2.8.5-src\hadoop-yarn-project\hadoop-yarn\hadoop-yarn-server\hadoop-yarn-server-nodemanager\src\main\java\org\apache\hadoop\yarn\server\nodemanager\containermanager\localizer\LocalizedResource.java
> log statement line number: 200
> log level:warn
> **/
> try {
>newState = this.stateMachine.doTransition(event.getType(), event);
> } catch (InvalidStateTransitionException e) {
>LOG.warn("Can't handle this event at current state", e);
> }
> /**
> file path:
> hadoop-2.8.5-src\hadoop-yarn-project\hadoop-yarn\hadoop-yarn-server\hadoop-yarn-server-nodemanager\src\main\java\org\apache\hadoop\yarn\server\nodemanager\containermanager\container\ContainerImpl.java
> log statement line number: 1156
> log level:warn
> **/
> try {
> newState =
> stateMachine.doTransition(event.getType(), event);
> } catch (InvalidStateTransitionException e) {
> LOG.warn("Can't handle this event at current state: Current: ["
> + oldState + "], eventType: [" + event.getType() + "]", e);
> }
> **ERROR level*
> /**
> file path:
> hadoop-2.8.5-src\hadoop-yarn-project\hadoop-yarn\hadoop-yarn-server\hadoop-yarn-server-resourcemanager\src\main\java\org\apache\hadoop\yarn\server\resourcemanager\rmapp\attempt\RMAppAttemptImpl.java
> log statement line number:878
> log level: error
> **/
> try {
>/* keep the master in sync with the state machine */
>this.stateMachine.doTransition(event.getType(), event);
> } catch (InvalidStateTransitionException e) {
>LOG.error("App attempt: " + appAttemptID
>+ " can't handle this event at current state", e);
>onInvalidTranstion(event.getType(), oldState);
> }
> /**
> file path:
> hadoop-2.8.5-src\hadoop-yarn-project\hadoop-yarn\hadoop-yarn-server\hadoop-yarn-server-resourcemanager\src\main\java\org\apache\hadoop\yarn\server\resourcemanager\rmnode\RMNodeImpl.java
> log statement line number:623
> log level: error
> **/
> try {
>stateMachine.doTransition(event.getType(), event);
> } catch (InvalidStateTransitionException e) {
>LOG.error("Can't handle this event at current state", e);
>LOG.error("Invalid event " + event.getType() +
>" on Node " + this.nodeId);
> }
>
> //There are 8 similar code snippets with ERROR log level.
> {code}
> After had a look on whole project, I found that there are 8 similar code
> snippets assgin the ERROR level, when doTransition() ocurrs
> *InvalidStateTransitionException*. And there are just 3 places choose the
> WARN level when in same situations. Therefor, I think these 3 log statements
> should
[jira] [Comment Edited] (YARN-9349) When doTransition() method occurs exception, the log level practices are inconsistent
[
https://issues.apache.org/jira/browse/YARN-9349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16819609#comment-16819609
]
Eric Yang edited comment on YARN-9349 at 4/16/19 11:56 PM:
---
+1 Thank you for the patch [~OneisAll]. I just committed this to trunk.
was (Author: eyang):
+1 Thank you for the patch [~OneisAll].
> When doTransition() method occurs exception, the log level practices are
> inconsistent
> -
>
> Key: YARN-9349
> URL: https://issues.apache.org/jira/browse/YARN-9349
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: nodemanager
>Affects Versions: 3.1.0, 2.8.5
>Reporter: Anuhan Torgonshar
>Priority: Major
> Labels: easyfix
> Fix For: 3.3.0
>
> Attachments: YARN-9349.trunk.patch
>
>
> There are *inconsistent* log level practices when code catches
> *_InvalidStateTransitionException_* for _*doTransition()*_ method.
> {code:java}
> **WARN level**
> /**
> file path:
> hadoop-2.8.5-src\hadoop-yarn-project\hadoop-yarn\hadoop-yarn-server\hadoop-yarn-server-nodemanager\src\main\java\org\apache\hadoop\yarn\server\nodemanager\containermanager\application\ApplicationImpl.java
> log statement line number: 482
> log level:warn
> **/
> try {
>// queue event requesting init of the same app
>newState = stateMachine.doTransition(event.getType(), event);
> } catch (InvalidStateTransitionException e) {
>LOG.warn("Can't handle this event at current state", e);
> }
> /**
> file path:
> hadoop-2.8.5-src\hadoop-yarn-project\hadoop-yarn\hadoop-yarn-server\hadoop-yarn-server-nodemanager\src\main\java\org\apache\hadoop\yarn\server\nodemanager\containermanager\localizer\LocalizedResource.java
> log statement line number: 200
> log level:warn
> **/
> try {
>newState = this.stateMachine.doTransition(event.getType(), event);
> } catch (InvalidStateTransitionException e) {
>LOG.warn("Can't handle this event at current state", e);
> }
> /**
> file path:
> hadoop-2.8.5-src\hadoop-yarn-project\hadoop-yarn\hadoop-yarn-server\hadoop-yarn-server-nodemanager\src\main\java\org\apache\hadoop\yarn\server\nodemanager\containermanager\container\ContainerImpl.java
> log statement line number: 1156
> log level:warn
> **/
> try {
> newState =
> stateMachine.doTransition(event.getType(), event);
> } catch (InvalidStateTransitionException e) {
> LOG.warn("Can't handle this event at current state: Current: ["
> + oldState + "], eventType: [" + event.getType() + "]", e);
> }
> **ERROR level*
> /**
> file path:
> hadoop-2.8.5-src\hadoop-yarn-project\hadoop-yarn\hadoop-yarn-server\hadoop-yarn-server-resourcemanager\src\main\java\org\apache\hadoop\yarn\server\resourcemanager\rmapp\attempt\RMAppAttemptImpl.java
> log statement line number:878
> log level: error
> **/
> try {
>/* keep the master in sync with the state machine */
>this.stateMachine.doTransition(event.getType(), event);
> } catch (InvalidStateTransitionException e) {
>LOG.error("App attempt: " + appAttemptID
>+ " can't handle this event at current state", e);
>onInvalidTranstion(event.getType(), oldState);
> }
> /**
> file path:
> hadoop-2.8.5-src\hadoop-yarn-project\hadoop-yarn\hadoop-yarn-server\hadoop-yarn-server-resourcemanager\src\main\java\org\apache\hadoop\yarn\server\resourcemanager\rmnode\RMNodeImpl.java
> log statement line number:623
> log level: error
> **/
> try {
>stateMachine.doTransition(event.getType(), event);
> } catch (InvalidStateTransitionException e) {
>LOG.error("Can't handle this event at current state", e);
>LOG.error("Invalid event " + event.getType() +
>" on Node " + this.nodeId);
> }
>
> //There are 8 similar code snippets with ERROR log level.
> {code}
> After had a look on whole project, I found that there are 8 similar code
> snippets assgin the ERROR level, when doTransition() ocurrs
> *InvalidStateTransitionException*. And there are just 3 places choose the
> WARN level when in same situations. Therefor, I think these 3 log statements
> should be assigned ERROR level to keep consistent with other code snippets.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6695) Race condition in RM for publishing container events vs appFinished events causes NPE
[
https://issues.apache.org/jira/browse/YARN-6695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16819603#comment-16819603
]
Eric Yang commented on YARN-6695:
-
[~Prabhu Joseph] I think another linger period flag is masking engineering
problem but not useful to system admin. Maybe we just log as debug when
NullPointerException is encountered. This will reduce the noise in log because
we know it's a race condition problem.
> Race condition in RM for publishing container events vs appFinished events
> causes NPE
> --
>
> Key: YARN-6695
> URL: https://issues.apache.org/jira/browse/YARN-6695
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Rohith Sharma K S
>Assignee: Prabhu Joseph
>Priority: Critical
> Attachments: YARN-6695.001.patch
>
>
> When RM publishes container events i.e by enabling
> *yarn.rm.system-metrics-publisher.emit-container-events*, there is race
> condition for processing events
> vs appFinished event that removes appId from collector list which cause NPE.
> Look at the below trace where appId is removed from collectors first and then
> corresponding events are processed.
> {noformat}
> 2017-06-06 19:28:48,896 INFO capacity.ParentQueue
> (ParentQueue.java:removeApplication(472)) - Application removed - appId:
> application_1496758895643_0005 user: root leaf-queue of parent: root
> #applications: 0
> 2017-06-06 19:28:48,921 INFO collector.TimelineCollectorManager
> (TimelineCollectorManager.java:remove(190)) - The collector service for
> application_1496758895643_0005 was removed
> 2017-06-06 19:28:48,922 ERROR metrics.TimelineServiceV2Publisher
> (TimelineServiceV2Publisher.java:putEntity(451)) - Error when publishing
> entity TimelineEntity[type='YARN_CONTAINER',
> id='container_e01_1496758895643_0005_01_02']
> java.lang.NullPointerException
> at
> org.apache.hadoop.yarn.server.resourcemanager.metrics.TimelineServiceV2Publisher.putEntity(TimelineServiceV2Publisher.java:448)
> at
> org.apache.hadoop.yarn.server.resourcemanager.metrics.TimelineServiceV2Publisher.access$100(TimelineServiceV2Publisher.java:72)
> at
> org.apache.hadoop.yarn.server.resourcemanager.metrics.TimelineServiceV2Publisher$TimelineV2EventHandler.handle(TimelineServiceV2Publisher.java:480)
> at
> org.apache.hadoop.yarn.server.resourcemanager.metrics.TimelineServiceV2Publisher$TimelineV2EventHandler.handle(TimelineServiceV2Publisher.java:469)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:201)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:127)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8551) Build Common module for MaWo application
[
https://issues.apache.org/jira/browse/YARN-8551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16819602#comment-16819602
]
Hadoop QA commented on YARN-8551:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
21s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
23s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
1s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 17m
3s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
17m 52s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-project hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
22s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
22s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 16m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 16m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
8s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 40s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-project hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-mawo
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
15s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
25s{color} | {color:green} hadoop-project in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 41m 1s{color}
| {color:red} hadoop-yarn-applications in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
31s{color} | {color:green} hadoop-yarn-applications-mawo in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
32s{color} | {color:green} hadoop-yarn-applications-mawo-core in the patch
passed. {color} |
| {color:green}+1{color} | {co
[jira] [Commented] (YARN-8622) NodeManager native build fails due to getgrouplist not found on macOS
[
https://issues.apache.org/jira/browse/YARN-8622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16819596#comment-16819596
]
Eric Yang commented on YARN-8622:
-
Can we use #ifdef __linux instead #ifdef __MACH? This will put Linux line
first to be consistent with other C code that have OS specific logic in
container-executor.
> NodeManager native build fails due to getgrouplist not found on macOS
> -
>
> Key: YARN-8622
> URL: https://issues.apache.org/jira/browse/YARN-8622
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
>Affects Versions: 3.2.0, 3.3.0
> Environment: Darwin 17.7.0 Darwin Kernel Version 17.7.0: Thu Jun 21
> 22:53:14 PDT 2018; root:xnu-4570.71.2~1/RELEASE_X86_64 x86_64
> Apple LLVM version 9.1.0 (clang-902.0.39.2)
>Reporter: Ewan Higgs
>Assignee: Siyao Meng
>Priority: Major
> Attachments: YARN-8622.001.patch
>
>
> Usage of getgrouplist() is added in YARN-7221 and should affect Hadoop 3.2.0
> and later.
> Compiler:
> {code}
> $ /Library/Developer/CommandLineTools/usr/bin/c++ --version
> Apple LLVM version 9.1.0 (clang-902.0.39.2)
> Target: x86_64-apple-darwin17.7.0
> Thread model: posix
> InstalledDir: /Library/Developer/CommandLineTools/usr/bin
> {code}
> Build line:
> {code}
> [WARNING] /Library/Developer/CommandLineTools/usr/bin/c++ -g -O2 -Wall
> -pthread -D_FILE_OFFSET_BITS=64 -Wl,-search_paths_first
> -Wl,-headerpad_max_install_names
> CMakeFiles/test-oom-listener.dir/main/native/oom-listener/impl/oom_listener.c.o
>
> CMakeFiles/test-oom-listener.dir/main/native/oom-listener/test/oom_listener_test_main.cc.o
> -o test/test-oom-listener libgtest.a -lrt
> {code}
> Error message:
> {code}
> ...
> [WARNING]
> /Users/ehiggs/src/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/utils/docker-util.c:1264:12:
> error: no matching function for call to 'getgrouplist'
> [WARNING] int rc = getgrouplist(user, pw->pw_gid, groups, &ngroups);
> [WARNING]^~~~
> [WARNING] /usr/include/unistd.h:653:6: note: candidate function not viable:
> no known conversion from 'gid_t *' (aka 'unsigned int *') to 'int *' for 3rd
> argument
> [WARNING] int getgrouplist(const char *, int, int *, int *);
> [WARNING] ^
> [WARNING] In file included from
> /Users/ehiggs/src/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/test/utils/test_docker_util.cc:24:
> [WARNING]
> /Users/ehiggs/src/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/utils/docker-util.c:1271:9:
> error: no matching function for call to 'getgrouplist'
> [WARNING] if (getgrouplist(user, pw->pw_gid, groups, &ngroups) == -1) {
> [WARNING] ^~~~
> [WARNING] /usr/include/unistd.h:653:6: note: candidate function not viable:
> no known conversion from 'gid_t *' (aka 'unsigned int *') to 'int *' for 3rd
> argument
> [WARNING] int getgrouplist(const char *, int, int *, int *);
> [WARNING] ^
> [WARNING] 2 warnings and 2 errors generated.
> [WARNING] make[2]: ***
> [CMakeFiles/cetest.dir/main/native/container-executor/test/utils/test_docker_util.cc.o]
> Error 1
> [WARNING] make[1]: *** [CMakeFiles/cetest.dir/all] Error 2
> [WARNING] make: *** [all] Error 2
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8551) Build Common module for MaWo application
[
https://issues.apache.org/jira/browse/YARN-8551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16819586#comment-16819586
]
Hadoop QA commented on YARN-8551:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
15s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
20s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
11s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
6s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
15m 21s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-project hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
34s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
25s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 14m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
9s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 4s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-project hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-mawo
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
5s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
28s{color} | {color:green} hadoop-project in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 41m 23s{color}
| {color:red} hadoop-yarn-applications in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
29s{color} | {color:green} hadoop-yarn-applications-mawo in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
28s{color} | {color:green} hadoop-yarn-applications-mawo-core in the patch
passed. {color} |
| {color:green}+1{color} | {co
[jira] [Commented] (YARN-9254) Externalize Solr data storage
[ https://issues.apache.org/jira/browse/YARN-9254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16819564#comment-16819564 ] Eric Yang commented on YARN-9254: - [~billie.rinaldi] Patch rebased to current trunk. Please review. > Externalize Solr data storage > - > > Key: YARN-9254 > URL: https://issues.apache.org/jira/browse/YARN-9254 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Eric Yang >Assignee: Eric Yang >Priority: Major > Attachments: YARN-9254.001.patch, YARN-9254.002.patch, > YARN-9254.003.patch, YARN-9254.004.patch, YARN-9254.005.patch > > > Application catalog contains embedded Solr. By default, Solr data is stored > in temp space of the docker container. For user who likes to persist Solr > data on HDFS, it would be nice to have a way to pass solr.hdfs.home setting > to embedded Solr to externalize Solr data storage. This also implies passing > Kerberos credential settings to Solr JVM in order to access secure HDFS. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8551) Build Common module for MaWo application
[ https://issues.apache.org/jira/browse/YARN-8551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yesha Vora updated YARN-8551: - Attachment: YARN-8551.0020.patch > Build Common module for MaWo application > > > Key: YARN-8551 > URL: https://issues.apache.org/jira/browse/YARN-8551 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Yesha Vora >Assignee: Yesha Vora >Priority: Major > Attachments: YARN-8551.001.patch, YARN-8551.0010.patch, > YARN-8551.0011.patch, YARN-8551.0012.patch, YARN-8551.0013.patch, > YARN-8551.0014.patch, YARN-8551.0015.patch, YARN-8551.0016.patch, > YARN-8551.0017.patch, YARN-8551.0018.patch, YARN-8551.0019.patch, > YARN-8551.002.patch, YARN-8551.0020.patch, YARN-8551.003.patch, > YARN-8551.004.patch, YARN-8551.005.patch, YARN-8551.006.patch, > YARN-8551.007.patch, YARN-8551.008.patch, YARN-8551.009.patch > > > Build Common module for MaWo application. > This module should include defination of Task. A Task should contain > * TaskID > * Task Command > * Task Environment > * Task Timeout > * Task Type > ** Simple Task > *** Its a single Task > ** Composite Task > *** Its a composition of multiple simple tasks > ** Teardown Task > *** Its a last task to be executed after a job is finished > ** Null Task > *** Its a null task -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8551) Build Common module for MaWo application
[ https://issues.apache.org/jira/browse/YARN-8551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yesha Vora updated YARN-8551: - Attachment: YARN-8551.0019.patch > Build Common module for MaWo application > > > Key: YARN-8551 > URL: https://issues.apache.org/jira/browse/YARN-8551 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Yesha Vora >Assignee: Yesha Vora >Priority: Major > Attachments: YARN-8551.001.patch, YARN-8551.0010.patch, > YARN-8551.0011.patch, YARN-8551.0012.patch, YARN-8551.0013.patch, > YARN-8551.0014.patch, YARN-8551.0015.patch, YARN-8551.0016.patch, > YARN-8551.0017.patch, YARN-8551.0018.patch, YARN-8551.0019.patch, > YARN-8551.002.patch, YARN-8551.003.patch, YARN-8551.004.patch, > YARN-8551.005.patch, YARN-8551.006.patch, YARN-8551.007.patch, > YARN-8551.008.patch, YARN-8551.009.patch > > > Build Common module for MaWo application. > This module should include defination of Task. A Task should contain > * TaskID > * Task Command > * Task Environment > * Task Timeout > * Task Type > ** Simple Task > *** Its a single Task > ** Composite Task > *** Its a composition of multiple simple tasks > ** Teardown Task > *** Its a last task to be executed after a job is finished > ** Null Task > *** Its a null task -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9123) Clean up and split testcases in TestNMWebServices for GPU support
[ https://issues.apache.org/jira/browse/YARN-9123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16819481#comment-16819481 ] Szilard Nemeth commented on YARN-9123: -- Oh I see. I didn't know this failure is coming up usually. > Clean up and split testcases in TestNMWebServices for GPU support > - > > Key: YARN-9123 > URL: https://issues.apache.org/jira/browse/YARN-9123 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Szilard Nemeth >Assignee: Szilard Nemeth >Priority: Minor > Fix For: 3.3.0 > > Attachments: YARN-9123.001.patch, YARN-9123.002.patch, > YARN-9123.003.patch, YARN-9123.004.patch, YARN-9123.005.patch, > YARN-9123.006.patch, YARN-9123.007.patch > > > The following testcases can be cleaned up a bit: > TestNMWebServices#testGetNMResourceInfo - Can be split up to 3 different cases > TestNMWebServices#testGetYarnGpuResourceInfo -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9202) RM does not track nodes that are in the include list and never register
[
https://issues.apache.org/jira/browse/YARN-9202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16819472#comment-16819472
]
Hadoop QA commented on YARN-9202:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 5s{color}
| {color:red} YARN-9202 does not apply to trunk. Rebase required? Wrong Branch?
See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | YARN-9202 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12955140/YARN-9202.001.patch |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/23976/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> RM does not track nodes that are in the include list and never register
> ---
>
> Key: YARN-9202
> URL: https://issues.apache.org/jira/browse/YARN-9202
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 2.9.2, 3.0.3, 2.8.5
>Reporter: Kuhu Shukla
>Assignee: Kuhu Shukla
>Priority: Major
> Attachments: YARN-9202.001.patch
>
>
> The RM state machine decides to put new or running nodes in inactive state
> only past the point of either registration or being in the exclude list. This
> does not cover the case where a node is the in the include list but never
> registers and since all state changes are based on these NodeState
> transitions, having NEW nodes be listed as inactive first may help. This
> would change the semantics of how inactiveNodes are looked at today. Another
> state addition might help this case too.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9123) Clean up and split testcases in TestNMWebServices for GPU support
[
https://issues.apache.org/jira/browse/YARN-9123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16819470#comment-16819470
]
Wei-Chiu Chuang commented on YARN-9123:
---
It's unrelated to this patch and Hudson always fails.
{noformat}
+ /home/jenkins/tools/maven/apache-maven-3.3.3/bin/mvn clean install
-DskipTests -Pnative --threads .5C
-Dmaven.repo.local=/home/jenkins/jenkins-slave/workspace/Hadoop-trunk-Commit/maven-repo
...
[ERROR] Failed to execute goal
org.apache.hadoop:hadoop-maven-plugins:3.3.0-SNAPSHOT:protoc (compile-protoc)
on project hadoop-common: org.apache.maven.plugin.MojoExecutionException:
protoc version is 'libprotoc 2.6.1', expected version is '2.5.0' -> [Help 1]
{noformat}
So environmental issue.
> Clean up and split testcases in TestNMWebServices for GPU support
> -
>
> Key: YARN-9123
> URL: https://issues.apache.org/jira/browse/YARN-9123
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: Szilard Nemeth
>Assignee: Szilard Nemeth
>Priority: Minor
> Fix For: 3.3.0
>
> Attachments: YARN-9123.001.patch, YARN-9123.002.patch,
> YARN-9123.003.patch, YARN-9123.004.patch, YARN-9123.005.patch,
> YARN-9123.006.patch, YARN-9123.007.patch
>
>
> The following testcases can be cleaned up a bit:
> TestNMWebServices#testGetNMResourceInfo - Can be split up to 3 different cases
> TestNMWebServices#testGetYarnGpuResourceInfo
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9202) RM does not track nodes that are in the include list and never register
[ https://issues.apache.org/jira/browse/YARN-9202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16819467#comment-16819467 ] Kuhu Shukla commented on YARN-9202: --- I do not think we can get away with creating new RMNodeImpl objects since anything that has not registered may not have valid values for cmPort and NmVersion and other fields that are populated through the constructor only upon registration. Even for the case where we could just have the REST APIs return state in new state, the issue is that none of the lists that the webservice has access to have nodes in new state. [~eepayne], appreciate thoughts on how to move forward on this given this inherent design of RMNodeImpl. I could expose some fields and add setters to get over this issue but I am not sure if that is the right way to proceed. > RM does not track nodes that are in the include list and never register > --- > > Key: YARN-9202 > URL: https://issues.apache.org/jira/browse/YARN-9202 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.9.2, 3.0.3, 2.8.5 >Reporter: Kuhu Shukla >Assignee: Kuhu Shukla >Priority: Major > Attachments: YARN-9202.001.patch > > > The RM state machine decides to put new or running nodes in inactive state > only past the point of either registration or being in the exclude list. This > does not cover the case where a node is the in the include list but never > registers and since all state changes are based on these NodeState > transitions, having NEW nodes be listed as inactive first may help. This > would change the semantics of how inactiveNodes are looked at today. Another > state addition might help this case too. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9475) Create basic VE plugin
[
https://issues.apache.org/jira/browse/YARN-9475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16819458#comment-16819458
]
Hadoop QA commented on YARN-9475:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
25s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
6s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 26s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
3s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
27s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
1s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 49s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 21m
4s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
26s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 75m 33s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:bdbca0e |
| JIRA Issue | YARN-9475 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12966131/YARN-9475-005.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 3da34ae23efe 4.4.0-144-generic #170~14.04.1-Ubuntu SMP Mon Mar
18 15:02:05 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / b8086ae |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/23975/testReport/ |
| Max. process+thread count | 305 (vs. ulimit of 1) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/23975/console |
| Powered by | Apache
[jira] [Commented] (YARN-9088) Non-exclusive labels break QueueMetrics
[ https://issues.apache.org/jira/browse/YARN-9088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16819455#comment-16819455 ] Karthik Palaniappan commented on YARN-9088: --- You'd also need to change how usedCapacity from YARN-6195 is calculated. It has similar logic for only the default partition. > Non-exclusive labels break QueueMetrics > --- > > Key: YARN-9088 > URL: https://issues.apache.org/jira/browse/YARN-9088 > Project: Hadoop YARN > Issue Type: Bug > Components: capacity scheduler, resourcemanager >Affects Versions: 2.8.5 >Reporter: Brandon Scheller >Priority: Major > Labels: metrics, nodelabel > > QueueMetrics are broken (random/negative values) when non-exclusive labels > are being used and unlabeled containers run on labeled nodes. > This is caused by the change in the patch here: > https://issues.apache.org/jira/browse/YARN-6467 > It assumes that a container's label will be the same as the node's label that > it is running on. > If you look within the patch, sometimes metrics are updated using the > request.getNodeLabelExpression(). And sometimes they are updated using > node.getPartition(). > This means that in the case where the node is labeled while the container > request isn't, these metrics only get updated when referring to the default > queue. This stops metrics from balancing out and results in incorrect and > negative values in QueueMetrics. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9088) Non-exclusive labels break QueueMetrics
[ https://issues.apache.org/jira/browse/YARN-9088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16819447#comment-16819447 ] Karthik Palaniappan commented on YARN-9088: --- +1. I think we should consider rolling back YARN-6467 instead of fixing it. I believe the original behavior was correct – metrics for the root queue should include metrics for all child queues and partitions. So AllocatedMB / AvailableMB, for example, give you a global view of cluster utilization. If YARN-6492 ever gets submitted, then we'll get per-partition metrics too. But I think YARN-6467 is the worst of both worlds – you don't get per partition metrics, and you don't get a global view of the cluster. A lot of cloud providers use cluster-level YARN metrics for autoscaling, and YARN-6467 breaks autoscaling. Side note: YARN-6467 was a breaking change with no documentation / release note. So rolling it back (another breaking change) should be fine. I'll attach a patch, as long as the rollback is straightforward. > Non-exclusive labels break QueueMetrics > --- > > Key: YARN-9088 > URL: https://issues.apache.org/jira/browse/YARN-9088 > Project: Hadoop YARN > Issue Type: Bug > Components: capacity scheduler, resourcemanager >Affects Versions: 2.8.5 >Reporter: Brandon Scheller >Priority: Major > Labels: metrics, nodelabel > > QueueMetrics are broken (random/negative values) when non-exclusive labels > are being used and unlabeled containers run on labeled nodes. > This is caused by the change in the patch here: > https://issues.apache.org/jira/browse/YARN-6467 > It assumes that a container's label will be the same as the node's label that > it is running on. > If you look within the patch, sometimes metrics are updated using the > request.getNodeLabelExpression(). And sometimes they are updated using > node.getPartition(). > This means that in the case where the node is labeled while the container > request isn't, these metrics only get updated when referring to the default > queue. This stops metrics from balancing out and results in incorrect and > negative values in QueueMetrics. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9339) Apps pending metric incorrect after moving app to a new queue
[
https://issues.apache.org/jira/browse/YARN-9339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16819445#comment-16819445
]
Hadoop QA commented on YARN-9339:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
57s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 56s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
12s{color} | {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
in trunk has 2 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
27s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 28s{color} | {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 1 new + 444 unchanged - 0 fixed = 445 total (was 444)
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 7s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 80m 29s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
29s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}128m 53s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestIncreaseAllocationExpirer
|
| |
hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairSchedulerPreemption |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:bdbca0e |
| JIRA Issue | YARN-9339 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12966120/YARN-9339.003.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux d61b8150f8fe 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24
10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / be6c801 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| findbugs | v3.1.0-RC1 |
| findbugs |
https://builds.apache.org/job/PreCommit-YARN-Build/23973/artifact/out/branch-findbugs-h
[jira] [Commented] (YARN-9123) Clean up and split testcases in TestNMWebServices for GPU support
[ https://issues.apache.org/jira/browse/YARN-9123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16819423#comment-16819423 ] Szilard Nemeth commented on YARN-9123: -- Thanks a lot [~jojochuang]! Do you know what caused the FAILURE coming from Jenkins in the above comment? Thanks! > Clean up and split testcases in TestNMWebServices for GPU support > - > > Key: YARN-9123 > URL: https://issues.apache.org/jira/browse/YARN-9123 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Szilard Nemeth >Assignee: Szilard Nemeth >Priority: Minor > Fix For: 3.3.0 > > Attachments: YARN-9123.001.patch, YARN-9123.002.patch, > YARN-9123.003.patch, YARN-9123.004.patch, YARN-9123.005.patch, > YARN-9123.006.patch, YARN-9123.007.patch > > > The following testcases can be cleaned up a bit: > TestNMWebServices#testGetNMResourceInfo - Can be split up to 3 different cases > TestNMWebServices#testGetYarnGpuResourceInfo -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9475) Create basic VE plugin
[ https://issues.apache.org/jira/browse/YARN-9475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Bacsko updated YARN-9475: --- Attachment: YARN-9475-005.patch > Create basic VE plugin > -- > > Key: YARN-9475 > URL: https://issues.apache.org/jira/browse/YARN-9475 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Reporter: Peter Bacsko >Assignee: Peter Bacsko >Priority: Major > Attachments: YARN-9475-001.patch, YARN-9475-002.patch, > YARN-9475-003.patch, YARN-9475-004.patch, YARN-9475-005.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9254) Externalize Solr data storage
[
https://issues.apache.org/jira/browse/YARN-9254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16819395#comment-16819395
]
Hadoop QA commented on YARN-9254:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
15s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
48s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
7s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 0s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
18s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} shellcheck {color} | {color:green} 0m
0s{color} | {color:green} There were no new shellcheck issues. {color} |
| {color:green}+1{color} | {color:green} shelldocs {color} | {color:green} 0m
17s{color} | {color:green} The patch generated 0 new + 104 unchanged - 132
fixed = 104 total (was 236) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 32s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
19s{color} | {color:green} hadoop-yarn-applications-catalog-docker in the patch
passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
16s{color} | {color:green} hadoop-yarn-site in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
30s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 43m 42s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:bdbca0e |
| JIRA Issue | YARN-9254 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12966121/YARN-9254.005.patch |
| Optional Tests | dupname asflicense mvnsite unit shellcheck shelldocs |
| uname | Linux a44fdb567f48 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24
10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / b8086ae |
| maven | version: Apache Maven 3.3.9 |
| shellcheck | v0.4.6 |
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/23974/testReport/ |
| Max. process+thread count | 447 (vs. ulimit of 1) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-catalog/hadoop-yarn-applications-catalog-docker
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site U:
hadoop-yarn-project/hadoop-yarn |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/23974/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> Externalize Solr data storage
> -
>
> Key: YARN-9254
> URL: https://issues.apache.org/jira/browse/YARN-9254
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Eric Yang
>Assignee: Eric Yang
>Priority: Major
>
[jira] [Commented] (YARN-9254) Externalize Solr data storage
[ https://issues.apache.org/jira/browse/YARN-9254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16819363#comment-16819363 ] Eric Yang commented on YARN-9254: - [~billie.rinaldi] Thank you for the review, patch 005 rebase to current trunk. > Externalize Solr data storage > - > > Key: YARN-9254 > URL: https://issues.apache.org/jira/browse/YARN-9254 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Eric Yang >Assignee: Eric Yang >Priority: Major > Attachments: YARN-9254.001.patch, YARN-9254.002.patch, > YARN-9254.003.patch, YARN-9254.004.patch, YARN-9254.005.patch > > > Application catalog contains embedded Solr. By default, Solr data is stored > in temp space of the docker container. For user who likes to persist Solr > data on HDFS, it would be nice to have a way to pass solr.hdfs.home setting > to embedded Solr to externalize Solr data storage. This also implies passing > Kerberos credential settings to Solr JVM in order to access secure HDFS. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9254) Externalize Solr data storage
[ https://issues.apache.org/jira/browse/YARN-9254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Yang updated YARN-9254: Attachment: YARN-9254.005.patch > Externalize Solr data storage > - > > Key: YARN-9254 > URL: https://issues.apache.org/jira/browse/YARN-9254 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Eric Yang >Assignee: Eric Yang >Priority: Major > Attachments: YARN-9254.001.patch, YARN-9254.002.patch, > YARN-9254.003.patch, YARN-9254.004.patch, YARN-9254.005.patch > > > Application catalog contains embedded Solr. By default, Solr data is stored > in temp space of the docker container. For user who likes to persist Solr > data on HDFS, it would be nice to have a way to pass solr.hdfs.home setting > to embedded Solr to externalize Solr data storage. This also implies passing > Kerberos credential settings to Solr JVM in order to access secure HDFS. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9123) Clean up and split testcases in TestNMWebServices for GPU support
[ https://issues.apache.org/jira/browse/YARN-9123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16819360#comment-16819360 ] Hudson commented on YARN-9123: -- FAILURE: Integrated in Jenkins build Hadoop-trunk-Commit #16421 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/16421/]) YARN-9123. Clean up and split testcases in TestNMWebServices for GPU (weichiu: rev b8086aed86ddf5bad19951b5ca2125369c882b8f) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/webapp/TestNMWebServices.java > Clean up and split testcases in TestNMWebServices for GPU support > - > > Key: YARN-9123 > URL: https://issues.apache.org/jira/browse/YARN-9123 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Szilard Nemeth >Assignee: Szilard Nemeth >Priority: Minor > Fix For: 3.3.0 > > Attachments: YARN-9123.001.patch, YARN-9123.002.patch, > YARN-9123.003.patch, YARN-9123.004.patch, YARN-9123.005.patch, > YARN-9123.006.patch, YARN-9123.007.patch > > > The following testcases can be cleaned up a bit: > TestNMWebServices#testGetNMResourceInfo - Can be split up to 3 different cases > TestNMWebServices#testGetYarnGpuResourceInfo -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8530) Add security filters to Application catalog
[ https://issues.apache.org/jira/browse/YARN-8530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16819358#comment-16819358 ] Eric Yang commented on YARN-8530: - Thanks for the review and commit [~billie.rinaldi]. > Add security filters to Application catalog > --- > > Key: YARN-8530 > URL: https://issues.apache.org/jira/browse/YARN-8530 > Project: Hadoop YARN > Issue Type: Sub-task > Components: security, yarn-native-services >Reporter: Eric Yang >Assignee: Eric Yang >Priority: Major > Fix For: 3.3.0 > > Attachments: YARN-8530.001.patch, YARN-8530.002.patch, > YARN-8530.003.patch, YARN-8530.004.patch, YARN-8530.005.patch > > > Application catalog UI does not have any security filter applied. CORS > filter and Authentication filter are required to secure the web application. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9123) Clean up and split testcases in TestNMWebServices for GPU support
[ https://issues.apache.org/jira/browse/YARN-9123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16819349#comment-16819349 ] Wei-Chiu Chuang commented on YARN-9123: --- +1 > Clean up and split testcases in TestNMWebServices for GPU support > - > > Key: YARN-9123 > URL: https://issues.apache.org/jira/browse/YARN-9123 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Szilard Nemeth >Assignee: Szilard Nemeth >Priority: Minor > Attachments: YARN-9123.001.patch, YARN-9123.002.patch, > YARN-9123.003.patch, YARN-9123.004.patch, YARN-9123.005.patch, > YARN-9123.006.patch, YARN-9123.007.patch > > > The following testcases can be cleaned up a bit: > TestNMWebServices#testGetNMResourceInfo - Can be split up to 3 different cases > TestNMWebServices#testGetYarnGpuResourceInfo -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9339) Apps pending metric incorrect after moving app to a new queue
[ https://issues.apache.org/jira/browse/YARN-9339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abhishek Modi updated YARN-9339: Attachment: YARN-9339.003.patch > Apps pending metric incorrect after moving app to a new queue > - > > Key: YARN-9339 > URL: https://issues.apache.org/jira/browse/YARN-9339 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Billie Rinaldi >Assignee: Abhishek Modi >Priority: Minor > Attachments: YARN-9339.001.patch, YARN-9339.002.patch, > YARN-9339.003.patch > > > I observed a cluster that had a high Apps Pending count that appeared to be > incorrect. This seemed to be related to apps being moved to different queues. > I tested by adding some logging to TestCapacityScheduler#testMoveAppBasic > before and after a moveApplication call. Before the call appsPending was 1 > and afterwards appsPending was 2. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8622) NodeManager native build fails due to getgrouplist not found on macOS
[
https://issues.apache.org/jira/browse/YARN-8622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16819329#comment-16819329
]
Wei-Chiu Chuang commented on YARN-8622:
---
+1 looks good to me.
[~eyang] FYI
> NodeManager native build fails due to getgrouplist not found on macOS
> -
>
> Key: YARN-8622
> URL: https://issues.apache.org/jira/browse/YARN-8622
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
>Affects Versions: 3.2.0, 3.3.0
> Environment: Darwin 17.7.0 Darwin Kernel Version 17.7.0: Thu Jun 21
> 22:53:14 PDT 2018; root:xnu-4570.71.2~1/RELEASE_X86_64 x86_64
> Apple LLVM version 9.1.0 (clang-902.0.39.2)
>Reporter: Ewan Higgs
>Assignee: Siyao Meng
>Priority: Major
> Attachments: YARN-8622.001.patch
>
>
> Usage of getgrouplist() is added in YARN-7221 and should affect Hadoop 3.2.0
> and later.
> Compiler:
> {code}
> $ /Library/Developer/CommandLineTools/usr/bin/c++ --version
> Apple LLVM version 9.1.0 (clang-902.0.39.2)
> Target: x86_64-apple-darwin17.7.0
> Thread model: posix
> InstalledDir: /Library/Developer/CommandLineTools/usr/bin
> {code}
> Build line:
> {code}
> [WARNING] /Library/Developer/CommandLineTools/usr/bin/c++ -g -O2 -Wall
> -pthread -D_FILE_OFFSET_BITS=64 -Wl,-search_paths_first
> -Wl,-headerpad_max_install_names
> CMakeFiles/test-oom-listener.dir/main/native/oom-listener/impl/oom_listener.c.o
>
> CMakeFiles/test-oom-listener.dir/main/native/oom-listener/test/oom_listener_test_main.cc.o
> -o test/test-oom-listener libgtest.a -lrt
> {code}
> Error message:
> {code}
> ...
> [WARNING]
> /Users/ehiggs/src/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/utils/docker-util.c:1264:12:
> error: no matching function for call to 'getgrouplist'
> [WARNING] int rc = getgrouplist(user, pw->pw_gid, groups, &ngroups);
> [WARNING]^~~~
> [WARNING] /usr/include/unistd.h:653:6: note: candidate function not viable:
> no known conversion from 'gid_t *' (aka 'unsigned int *') to 'int *' for 3rd
> argument
> [WARNING] int getgrouplist(const char *, int, int *, int *);
> [WARNING] ^
> [WARNING] In file included from
> /Users/ehiggs/src/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/test/utils/test_docker_util.cc:24:
> [WARNING]
> /Users/ehiggs/src/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/utils/docker-util.c:1271:9:
> error: no matching function for call to 'getgrouplist'
> [WARNING] if (getgrouplist(user, pw->pw_gid, groups, &ngroups) == -1) {
> [WARNING] ^~~~
> [WARNING] /usr/include/unistd.h:653:6: note: candidate function not viable:
> no known conversion from 'gid_t *' (aka 'unsigned int *') to 'int *' for 3rd
> argument
> [WARNING] int getgrouplist(const char *, int, int *, int *);
> [WARNING] ^
> [WARNING] 2 warnings and 2 errors generated.
> [WARNING] make[2]: ***
> [CMakeFiles/cetest.dir/main/native/container-executor/test/utils/test_docker_util.cc.o]
> Error 1
> [WARNING] make[1]: *** [CMakeFiles/cetest.dir/all] Error 2
> [WARNING] make: *** [all] Error 2
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8530) Add security filters to Application catalog
[ https://issues.apache.org/jira/browse/YARN-8530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16819328#comment-16819328 ] Billie Rinaldi commented on YARN-8530: -- Looks like this failure was due to protoc version: protoc version is 'libprotoc 2.6.1', expected version is '2.5.0'. > Add security filters to Application catalog > --- > > Key: YARN-8530 > URL: https://issues.apache.org/jira/browse/YARN-8530 > Project: Hadoop YARN > Issue Type: Sub-task > Components: security, yarn-native-services >Reporter: Eric Yang >Assignee: Eric Yang >Priority: Major > Fix For: 3.3.0 > > Attachments: YARN-8530.001.patch, YARN-8530.002.patch, > YARN-8530.003.patch, YARN-8530.004.patch, YARN-8530.005.patch > > > Application catalog UI does not have any security filter applied. CORS > filter and Authentication filter are required to secure the web application. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9487) NodeManager native build shouldn't link against librt on macOS
[
https://issues.apache.org/jira/browse/YARN-9487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16819327#comment-16819327
]
Wei-Chiu Chuang commented on YARN-9487:
---
+1
I think it'll also be useful if you could update the instruction for building
Hadoop native lib on Mac.
> NodeManager native build shouldn't link against librt on macOS
> --
>
> Key: YARN-9487
> URL: https://issues.apache.org/jira/browse/YARN-9487
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
>Affects Versions: 3.2.0, 3.3.0
> Environment: Darwin 18.5.0 Darwin Kernel Version 18.5.0: Mon Mar 11
> 20:40:32 PDT 2019; root:xnu-4903.251.3~3/RELEASE_X86_64 x86_64
> Apple LLVM version 10.0.1 (clang-1001.0.46.3)
>Reporter: Siyao Meng
>Assignee: Siyao Meng
>Priority: Major
> Attachments: YARN-9487.001.patch
>
>
> I'm compiling hadoop/trunk native on a Mac. Found that when compiling
> NodeManager it is trying to link against librt on Darwin as well - and it
> failed for sure. Diff for a quick fix:
> {code}
> diff --git
> a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/CMakeLists.txt
>
> b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/CMakeLists.txt
> index f0f005d53b5..9c3f0135aaf 100644
> ---
> a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/CMakeLists.txt
> +++
> b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/CMakeLists.txt
> @@ -190,5 +190,10 @@ add_executable(test-oom-listener
> main/native/oom-listener/impl/oom_listener.h
> main/native/oom-listener/test/oom_listener_test_main.cc
> )
> -target_link_libraries(test-oom-listener gtest rt)
> +if(CMAKE_SYSTEM_NAME MATCHES "Darwin")
> +# macOS does not have rt
> +target_link_libraries(test-oom-listener gtest)
> +else()
> +target_link_libraries(test-oom-listener gtest rt)
> +endif()
> output_directory(test-oom-listener test)
> {code}
> This line of code is added in YARN-8437, so this affects 3.2.0+.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8622) NodeManager native build fails due to getgrouplist not found on macOS
[
https://issues.apache.org/jira/browse/YARN-8622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16819308#comment-16819308
]
Hadoop QA commented on YARN-8622:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
4s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
29m 32s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 30s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 21m
20s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
28s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 66m 53s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:bdbca0e |
| JIRA Issue | YARN-8622 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12966111/YARN-8622.001.patch |
| Optional Tests | dupname asflicense compile cc mvnsite javac unit |
| uname | Linux d363401db067 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24
10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / bfcb653 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/23972/testReport/ |
| Max. process+thread count | 446 (vs. ulimit of 1) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/23972/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> NodeManager native build fails due to getgrouplist not found on macOS
> -
>
> Key: YARN-8622
> URL: https://issues.apache.org/jira/browse/YARN-8622
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
>Affects Versions: 3.2.0, 3.3.0
> Environment: Darwin 17.7.0 Darwin Kernel Version 17.7.0: Thu Jun 21
> 22:53:14 PDT 2018; root:xnu-4570.71.2~1/RELEASE_X86_64 x86_64
> Apple LLVM version 9.1.0 (clang-902.0.39.2)
>Reporter: Ewan Higgs
>Assignee: Siyao Meng
>Priority:
[jira] [Commented] (YARN-9254) Externalize Solr data storage
[ https://issues.apache.org/jira/browse/YARN-9254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16819284#comment-16819284 ] Billie Rinaldi commented on YARN-9254: -- Thanks for the patch, [~eyang]! It looks like this one has a conflict. Please rebase. > Externalize Solr data storage > - > > Key: YARN-9254 > URL: https://issues.apache.org/jira/browse/YARN-9254 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Eric Yang >Assignee: Eric Yang >Priority: Major > Attachments: YARN-9254.001.patch, YARN-9254.002.patch, > YARN-9254.003.patch, YARN-9254.004.patch > > > Application catalog contains embedded Solr. By default, Solr data is stored > in temp space of the docker container. For user who likes to persist Solr > data on HDFS, it would be nice to have a way to pass solr.hdfs.home setting > to embedded Solr to externalize Solr data storage. This also implies passing > Kerberos credential settings to Solr JVM in order to access secure HDFS. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9466) App catalog navigation stylesheet does not display correctly in Safari
[ https://issues.apache.org/jira/browse/YARN-9466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16819280#comment-16819280 ] Hudson commented on YARN-9466: -- FAILURE: Integrated in Jenkins build Hadoop-trunk-Commit #16419 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/16419/]) YARN-9466. Fixed application catalog navigation bar height in Safari. (billie: rev 2364c7d0bf22f042b6a564b863fcfdbac48c4bfb) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-catalog/hadoop-yarn-applications-catalog-webapp/src/main/webapp/css/specific.css * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-catalog/hadoop-yarn-applications-catalog-webapp/src/main/webapp/index.html * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-catalog/hadoop-yarn-applications-catalog-webapp/src/main/webapp/partials/home.html * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-catalog/hadoop-yarn-applications-catalog-webapp/src/main/webapp/css/bootstrap-hadoop.css > App catalog navigation stylesheet does not display correctly in Safari > -- > > Key: YARN-9466 > URL: https://issues.apache.org/jira/browse/YARN-9466 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Eric Yang >Assignee: Eric Yang >Priority: Major > Fix For: 3.3.0 > > Attachments: YARN-9466.001.patch, YARN-9466.002.patch, > catalog-chrome.png, catalog-safari.png > > > When navigation side bar has less content than right side table, the > navigation bar will shrink into smaller size in Safari. See the attached > screenshot for the problem and desired looked. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9466) App catalog navigation stylesheet does not display correctly in Safari
[ https://issues.apache.org/jira/browse/YARN-9466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16819276#comment-16819276 ] Billie Rinaldi commented on YARN-9466: -- +1 for patch 2. This appears to fix the Safari issue. Thanks [~eyang]! > App catalog navigation stylesheet does not display correctly in Safari > -- > > Key: YARN-9466 > URL: https://issues.apache.org/jira/browse/YARN-9466 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Eric Yang >Assignee: Eric Yang >Priority: Major > Attachments: YARN-9466.001.patch, YARN-9466.002.patch, > catalog-chrome.png, catalog-safari.png > > > When navigation side bar has less content than right side table, the > navigation bar will shrink into smaller size in Safari. See the attached > screenshot for the problem and desired looked. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8530) Add security filters to Application catalog
[ https://issues.apache.org/jira/browse/YARN-8530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16819264#comment-16819264 ] Hudson commented on YARN-8530: -- FAILURE: Integrated in Jenkins build Hadoop-trunk-Commit #16418 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/16418/]) YARN-8530. Add SPNEGO filter to application catalog. Contributed by Eric (billie: rev ad865888a6980d1f1bd6c56dbac1252aeed8091b) * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-catalog/hadoop-yarn-applications-catalog-webapp/src/main/java/org/apache/hadoop/yarn/appcatalog/application/AppCatalogInitializer.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-catalog/hadoop-yarn-applications-catalog-webapp/src/main/webapp/WEB-INF/web.xml * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/yarn-service/Examples.md * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-catalog/hadoop-yarn-applications-catalog-docker/src/main/scripts/entrypoint.sh * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-catalog/hadoop-yarn-applications-catalog-docker/src/main/scripts/setup-image.sh > Add security filters to Application catalog > --- > > Key: YARN-8530 > URL: https://issues.apache.org/jira/browse/YARN-8530 > Project: Hadoop YARN > Issue Type: Sub-task > Components: security, yarn-native-services >Reporter: Eric Yang >Assignee: Eric Yang >Priority: Major > Fix For: 3.3.0 > > Attachments: YARN-8530.001.patch, YARN-8530.002.patch, > YARN-8530.003.patch, YARN-8530.004.patch, YARN-8530.005.patch > > > Application catalog UI does not have any security filter applied. CORS > filter and Authentication filter are required to secure the web application. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8530) Add security filters to Application catalog
[ https://issues.apache.org/jira/browse/YARN-8530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16819251#comment-16819251 ] Billie Rinaldi commented on YARN-8530: -- +1 for patch 5. Thanks [~eyang]! > Add security filters to Application catalog > --- > > Key: YARN-8530 > URL: https://issues.apache.org/jira/browse/YARN-8530 > Project: Hadoop YARN > Issue Type: Sub-task > Components: security, yarn-native-services >Reporter: Eric Yang >Assignee: Eric Yang >Priority: Major > Attachments: YARN-8530.001.patch, YARN-8530.002.patch, > YARN-8530.003.patch, YARN-8530.004.patch, YARN-8530.005.patch > > > Application catalog UI does not have any security filter applied. CORS > filter and Authentication filter are required to secure the web application. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8622) NodeManager native build fails due to getgrouplist not found on macOS
[
https://issues.apache.org/jira/browse/YARN-8622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated YARN-8622:
-
Summary: NodeManager native build fails due to getgrouplist not found on
macOS (was: Native build fails on macos due to getgrouplist not found)
> NodeManager native build fails due to getgrouplist not found on macOS
> -
>
> Key: YARN-8622
> URL: https://issues.apache.org/jira/browse/YARN-8622
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
>Affects Versions: 3.2.0, 3.3.0
> Environment: Darwin 17.7.0 Darwin Kernel Version 17.7.0: Thu Jun 21
> 22:53:14 PDT 2018; root:xnu-4570.71.2~1/RELEASE_X86_64 x86_64
> Apple LLVM version 9.1.0 (clang-902.0.39.2)
>Reporter: Ewan Higgs
>Assignee: Siyao Meng
>Priority: Major
> Attachments: YARN-8622.001.patch
>
>
> Usage of getgrouplist() is added in YARN-7221 and should affect Hadoop 3.2.0
> and later.
> Compiler:
> {code}
> $ /Library/Developer/CommandLineTools/usr/bin/c++ --version
> Apple LLVM version 9.1.0 (clang-902.0.39.2)
> Target: x86_64-apple-darwin17.7.0
> Thread model: posix
> InstalledDir: /Library/Developer/CommandLineTools/usr/bin
> {code}
> Build line:
> {code}
> [WARNING] /Library/Developer/CommandLineTools/usr/bin/c++ -g -O2 -Wall
> -pthread -D_FILE_OFFSET_BITS=64 -Wl,-search_paths_first
> -Wl,-headerpad_max_install_names
> CMakeFiles/test-oom-listener.dir/main/native/oom-listener/impl/oom_listener.c.o
>
> CMakeFiles/test-oom-listener.dir/main/native/oom-listener/test/oom_listener_test_main.cc.o
> -o test/test-oom-listener libgtest.a -lrt
> {code}
> Error message:
> {code}
> ...
> [WARNING]
> /Users/ehiggs/src/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/utils/docker-util.c:1264:12:
> error: no matching function for call to 'getgrouplist'
> [WARNING] int rc = getgrouplist(user, pw->pw_gid, groups, &ngroups);
> [WARNING]^~~~
> [WARNING] /usr/include/unistd.h:653:6: note: candidate function not viable:
> no known conversion from 'gid_t *' (aka 'unsigned int *') to 'int *' for 3rd
> argument
> [WARNING] int getgrouplist(const char *, int, int *, int *);
> [WARNING] ^
> [WARNING] In file included from
> /Users/ehiggs/src/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/test/utils/test_docker_util.cc:24:
> [WARNING]
> /Users/ehiggs/src/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/utils/docker-util.c:1271:9:
> error: no matching function for call to 'getgrouplist'
> [WARNING] if (getgrouplist(user, pw->pw_gid, groups, &ngroups) == -1) {
> [WARNING] ^~~~
> [WARNING] /usr/include/unistd.h:653:6: note: candidate function not viable:
> no known conversion from 'gid_t *' (aka 'unsigned int *') to 'int *' for 3rd
> argument
> [WARNING] int getgrouplist(const char *, int, int *, int *);
> [WARNING] ^
> [WARNING] 2 warnings and 2 errors generated.
> [WARNING] make[2]: ***
> [CMakeFiles/cetest.dir/main/native/container-executor/test/utils/test_docker_util.cc.o]
> Error 1
> [WARNING] make[1]: *** [CMakeFiles/cetest.dir/all] Error 2
> [WARNING] make: *** [all] Error 2
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9487) NodeManager native build shouldn't link against librt on macOS
[
https://issues.apache.org/jira/browse/YARN-9487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated YARN-9487:
-
Summary: NodeManager native build shouldn't link against librt on macOS
(was: NodeManager shouldn't link against librt on macOS)
> NodeManager native build shouldn't link against librt on macOS
> --
>
> Key: YARN-9487
> URL: https://issues.apache.org/jira/browse/YARN-9487
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
>Affects Versions: 3.2.0, 3.3.0
> Environment: Darwin 18.5.0 Darwin Kernel Version 18.5.0: Mon Mar 11
> 20:40:32 PDT 2019; root:xnu-4903.251.3~3/RELEASE_X86_64 x86_64
> Apple LLVM version 10.0.1 (clang-1001.0.46.3)
>Reporter: Siyao Meng
>Assignee: Siyao Meng
>Priority: Major
> Attachments: YARN-9487.001.patch
>
>
> I'm compiling hadoop/trunk native on a Mac. Found that when compiling
> NodeManager it is trying to link against librt on Darwin as well - and it
> failed for sure. Diff for a quick fix:
> {code}
> diff --git
> a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/CMakeLists.txt
>
> b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/CMakeLists.txt
> index f0f005d53b5..9c3f0135aaf 100644
> ---
> a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/CMakeLists.txt
> +++
> b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/CMakeLists.txt
> @@ -190,5 +190,10 @@ add_executable(test-oom-listener
> main/native/oom-listener/impl/oom_listener.h
> main/native/oom-listener/test/oom_listener_test_main.cc
> )
> -target_link_libraries(test-oom-listener gtest rt)
> +if(CMAKE_SYSTEM_NAME MATCHES "Darwin")
> +# macOS does not have rt
> +target_link_libraries(test-oom-listener gtest)
> +else()
> +target_link_libraries(test-oom-listener gtest rt)
> +endif()
> output_directory(test-oom-listener test)
> {code}
> This line of code is added in YARN-8437, so this affects 3.2.0+.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9487) NodeManager shouldn't link against librt on macOS
[
https://issues.apache.org/jira/browse/YARN-9487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated YARN-9487:
-
Environment:
Darwin 18.5.0 Darwin Kernel Version 18.5.0: Mon Mar 11 20:40:32 PDT 2019;
root:xnu-4903.251.3~3/RELEASE_X86_64 x86_64
Apple LLVM version 10.0.1 (clang-1001.0.46.3)
> NodeManager shouldn't link against librt on macOS
> -
>
> Key: YARN-9487
> URL: https://issues.apache.org/jira/browse/YARN-9487
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
>Affects Versions: 3.2.0, 3.3.0
> Environment: Darwin 18.5.0 Darwin Kernel Version 18.5.0: Mon Mar 11
> 20:40:32 PDT 2019; root:xnu-4903.251.3~3/RELEASE_X86_64 x86_64
> Apple LLVM version 10.0.1 (clang-1001.0.46.3)
>Reporter: Siyao Meng
>Assignee: Siyao Meng
>Priority: Major
> Attachments: YARN-9487.001.patch
>
>
> I'm compiling hadoop/trunk native on a Mac. Found that when compiling
> NodeManager it is trying to link against librt on Darwin as well - and it
> failed for sure. Diff for a quick fix:
> {code}
> diff --git
> a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/CMakeLists.txt
>
> b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/CMakeLists.txt
> index f0f005d53b5..9c3f0135aaf 100644
> ---
> a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/CMakeLists.txt
> +++
> b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/CMakeLists.txt
> @@ -190,5 +190,10 @@ add_executable(test-oom-listener
> main/native/oom-listener/impl/oom_listener.h
> main/native/oom-listener/test/oom_listener_test_main.cc
> )
> -target_link_libraries(test-oom-listener gtest rt)
> +if(CMAKE_SYSTEM_NAME MATCHES "Darwin")
> +# macOS does not have rt
> +target_link_libraries(test-oom-listener gtest)
> +else()
> +target_link_libraries(test-oom-listener gtest rt)
> +endif()
> output_directory(test-oom-listener test)
> {code}
> This line of code is added in YARN-8437, so this affects 3.2.0+.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8622) Native build fails on macos due to getgrouplist not found
[
https://issues.apache.org/jira/browse/YARN-8622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated YARN-8622:
-
Description:
Usage of getgrouplist() is added in YARN-7221 and should affect Hadoop 3.2.0
and later.
Compiler:
{code}
$ /Library/Developer/CommandLineTools/usr/bin/c++ --version
Apple LLVM version 9.1.0 (clang-902.0.39.2)
Target: x86_64-apple-darwin17.7.0
Thread model: posix
InstalledDir: /Library/Developer/CommandLineTools/usr/bin
{code}
Build line:
{code}
[WARNING] /Library/Developer/CommandLineTools/usr/bin/c++ -g -O2 -Wall
-pthread -D_FILE_OFFSET_BITS=64 -Wl,-search_paths_first
-Wl,-headerpad_max_install_names
CMakeFiles/test-oom-listener.dir/main/native/oom-listener/impl/oom_listener.c.o
CMakeFiles/test-oom-listener.dir/main/native/oom-listener/test/oom_listener_test_main.cc.o
-o test/test-oom-listener libgtest.a -lrt
{code}
Error message:
{code}
...
[WARNING]
/Users/ehiggs/src/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/utils/docker-util.c:1264:12:
error: no matching function for call to 'getgrouplist'
[WARNING] int rc = getgrouplist(user, pw->pw_gid, groups, &ngroups);
[WARNING]^~~~
[WARNING] /usr/include/unistd.h:653:6: note: candidate function not viable: no
known conversion from 'gid_t *' (aka 'unsigned int *') to 'int *' for 3rd
argument
[WARNING] int getgrouplist(const char *, int, int *, int *);
[WARNING] ^
[WARNING] In file included from
/Users/ehiggs/src/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/test/utils/test_docker_util.cc:24:
[WARNING]
/Users/ehiggs/src/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/utils/docker-util.c:1271:9:
error: no matching function for call to 'getgrouplist'
[WARNING] if (getgrouplist(user, pw->pw_gid, groups, &ngroups) == -1) {
[WARNING] ^~~~
[WARNING] /usr/include/unistd.h:653:6: note: candidate function not viable: no
known conversion from 'gid_t *' (aka 'unsigned int *') to 'int *' for 3rd
argument
[WARNING] int getgrouplist(const char *, int, int *, int *);
[WARNING] ^
[WARNING] 2 warnings and 2 errors generated.
[WARNING] make[2]: ***
[CMakeFiles/cetest.dir/main/native/container-executor/test/utils/test_docker_util.cc.o]
Error 1
[WARNING] make[1]: *** [CMakeFiles/cetest.dir/all] Error 2
[WARNING] make: *** [all] Error 2
{code}
was:
Compiler:
{code}
$ /Library/Developer/CommandLineTools/usr/bin/c++ --version
Apple LLVM version 9.1.0 (clang-902.0.39.2)
Target: x86_64-apple-darwin17.7.0
Thread model: posix
InstalledDir: /Library/Developer/CommandLineTools/usr/bin
{code}
Build line:
{code}
[WARNING] /Library/Developer/CommandLineTools/usr/bin/c++ -g -O2 -Wall
-pthread -D_FILE_OFFSET_BITS=64 -Wl,-search_paths_first
-Wl,-headerpad_max_install_names
CMakeFiles/test-oom-listener.dir/main/native/oom-listener/impl/oom_listener.c.o
CMakeFiles/test-oom-listener.dir/main/native/oom-listener/test/oom_listener_test_main.cc.o
-o test/test-oom-listener libgtest.a -lrt
{code}
Error message:
{code}
...
[WARNING]
/Users/ehiggs/src/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/utils/docker-util.c:1264:12:
error: no matching function for call to 'getgrouplist'
[WARNING] int rc = getgrouplist(user, pw->pw_gid, groups, &ngroups);
[WARNING]^~~~
[WARNING] /usr/include/unistd.h:653:6: note: candidate function not viable: no
known conversion from 'gid_t *' (aka 'unsigned int *') to 'int *' for 3rd
argument
[WARNING] int getgrouplist(const char *, int, int *, int *);
[WARNING] ^
[WARNING] In file included from
/Users/ehiggs/src/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/test/utils/test_docker_util.cc:24:
[WARNING]
/Users/ehiggs/src/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/utils/docker-util.c:1271:9:
error: no matching function for call to 'getgrouplist'
[WARNING] if (getgrouplist(user, pw->pw_gid, groups, &ngroups) == -1) {
[WARNING] ^~~~
[WARNING] /usr/include/unistd.h:653:6: note: candidate function not viable: no
known conversion from 'gid_t *' (aka 'unsigned int *') to 'int *' for 3rd
argument
[WARNING] int getgrouplist(const char *, int, int *, int *);
[WARNING] ^
[WARNING] 2 warnings and 2 errors generated.
[WARNING] make[2]: ***
[CMakeFiles/cetest.dir/main/native/container-executor/test/utils/test_docker_util.cc.o]
Error 1
[WARNING] make[1]: *** [CMakeFiles/cetest.dir/all] Error 2
[WARNING] make: *** [all]
[jira] [Updated] (YARN-8622) Native build fails on macos due to getgrouplist not found
[
https://issues.apache.org/jira/browse/YARN-8622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated YARN-8622:
-
Affects Version/s: 3.2.0
> Native build fails on macos due to getgrouplist not found
> -
>
> Key: YARN-8622
> URL: https://issues.apache.org/jira/browse/YARN-8622
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 3.2.0
> Environment: Darwin 17.7.0 Darwin Kernel Version 17.7.0: Thu Jun 21
> 22:53:14 PDT 2018; root:xnu-4570.71.2~1/RELEASE_X86_64 x86_64
> Apple LLVM version 9.1.0 (clang-902.0.39.2)
>Reporter: Ewan Higgs
>Assignee: Siyao Meng
>Priority: Major
> Attachments: YARN-8622.001.patch
>
>
> Usage of getgrouplist() is added in YARN-7221 and should affect Hadoop 3.2.0
> and later.
> Compiler:
> {code}
> $ /Library/Developer/CommandLineTools/usr/bin/c++ --version
> Apple LLVM version 9.1.0 (clang-902.0.39.2)
> Target: x86_64-apple-darwin17.7.0
> Thread model: posix
> InstalledDir: /Library/Developer/CommandLineTools/usr/bin
> {code}
> Build line:
> {code}
> [WARNING] /Library/Developer/CommandLineTools/usr/bin/c++ -g -O2 -Wall
> -pthread -D_FILE_OFFSET_BITS=64 -Wl,-search_paths_first
> -Wl,-headerpad_max_install_names
> CMakeFiles/test-oom-listener.dir/main/native/oom-listener/impl/oom_listener.c.o
>
> CMakeFiles/test-oom-listener.dir/main/native/oom-listener/test/oom_listener_test_main.cc.o
> -o test/test-oom-listener libgtest.a -lrt
> {code}
> Error message:
> {code}
> ...
> [WARNING]
> /Users/ehiggs/src/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/utils/docker-util.c:1264:12:
> error: no matching function for call to 'getgrouplist'
> [WARNING] int rc = getgrouplist(user, pw->pw_gid, groups, &ngroups);
> [WARNING]^~~~
> [WARNING] /usr/include/unistd.h:653:6: note: candidate function not viable:
> no known conversion from 'gid_t *' (aka 'unsigned int *') to 'int *' for 3rd
> argument
> [WARNING] int getgrouplist(const char *, int, int *, int *);
> [WARNING] ^
> [WARNING] In file included from
> /Users/ehiggs/src/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/test/utils/test_docker_util.cc:24:
> [WARNING]
> /Users/ehiggs/src/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/utils/docker-util.c:1271:9:
> error: no matching function for call to 'getgrouplist'
> [WARNING] if (getgrouplist(user, pw->pw_gid, groups, &ngroups) == -1) {
> [WARNING] ^~~~
> [WARNING] /usr/include/unistd.h:653:6: note: candidate function not viable:
> no known conversion from 'gid_t *' (aka 'unsigned int *') to 'int *' for 3rd
> argument
> [WARNING] int getgrouplist(const char *, int, int *, int *);
> [WARNING] ^
> [WARNING] 2 warnings and 2 errors generated.
> [WARNING] make[2]: ***
> [CMakeFiles/cetest.dir/main/native/container-executor/test/utils/test_docker_util.cc.o]
> Error 1
> [WARNING] make[1]: *** [CMakeFiles/cetest.dir/all] Error 2
> [WARNING] make: *** [all] Error 2
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8622) Native build fails on macos due to getgrouplist not found
[
https://issues.apache.org/jira/browse/YARN-8622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated YARN-8622:
-
Component/s: nodemanager
> Native build fails on macos due to getgrouplist not found
> -
>
> Key: YARN-8622
> URL: https://issues.apache.org/jira/browse/YARN-8622
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
>Affects Versions: 3.2.0
> Environment: Darwin 17.7.0 Darwin Kernel Version 17.7.0: Thu Jun 21
> 22:53:14 PDT 2018; root:xnu-4570.71.2~1/RELEASE_X86_64 x86_64
> Apple LLVM version 9.1.0 (clang-902.0.39.2)
>Reporter: Ewan Higgs
>Assignee: Siyao Meng
>Priority: Major
> Attachments: YARN-8622.001.patch
>
>
> Usage of getgrouplist() is added in YARN-7221 and should affect Hadoop 3.2.0
> and later.
> Compiler:
> {code}
> $ /Library/Developer/CommandLineTools/usr/bin/c++ --version
> Apple LLVM version 9.1.0 (clang-902.0.39.2)
> Target: x86_64-apple-darwin17.7.0
> Thread model: posix
> InstalledDir: /Library/Developer/CommandLineTools/usr/bin
> {code}
> Build line:
> {code}
> [WARNING] /Library/Developer/CommandLineTools/usr/bin/c++ -g -O2 -Wall
> -pthread -D_FILE_OFFSET_BITS=64 -Wl,-search_paths_first
> -Wl,-headerpad_max_install_names
> CMakeFiles/test-oom-listener.dir/main/native/oom-listener/impl/oom_listener.c.o
>
> CMakeFiles/test-oom-listener.dir/main/native/oom-listener/test/oom_listener_test_main.cc.o
> -o test/test-oom-listener libgtest.a -lrt
> {code}
> Error message:
> {code}
> ...
> [WARNING]
> /Users/ehiggs/src/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/utils/docker-util.c:1264:12:
> error: no matching function for call to 'getgrouplist'
> [WARNING] int rc = getgrouplist(user, pw->pw_gid, groups, &ngroups);
> [WARNING]^~~~
> [WARNING] /usr/include/unistd.h:653:6: note: candidate function not viable:
> no known conversion from 'gid_t *' (aka 'unsigned int *') to 'int *' for 3rd
> argument
> [WARNING] int getgrouplist(const char *, int, int *, int *);
> [WARNING] ^
> [WARNING] In file included from
> /Users/ehiggs/src/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/test/utils/test_docker_util.cc:24:
> [WARNING]
> /Users/ehiggs/src/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/utils/docker-util.c:1271:9:
> error: no matching function for call to 'getgrouplist'
> [WARNING] if (getgrouplist(user, pw->pw_gid, groups, &ngroups) == -1) {
> [WARNING] ^~~~
> [WARNING] /usr/include/unistd.h:653:6: note: candidate function not viable:
> no known conversion from 'gid_t *' (aka 'unsigned int *') to 'int *' for 3rd
> argument
> [WARNING] int getgrouplist(const char *, int, int *, int *);
> [WARNING] ^
> [WARNING] 2 warnings and 2 errors generated.
> [WARNING] make[2]: ***
> [CMakeFiles/cetest.dir/main/native/container-executor/test/utils/test_docker_util.cc.o]
> Error 1
> [WARNING] make[1]: *** [CMakeFiles/cetest.dir/all] Error 2
> [WARNING] make: *** [all] Error 2
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8622) Native build fails on macos due to getgrouplist not found
[
https://issues.apache.org/jira/browse/YARN-8622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated YARN-8622:
-
Affects Version/s: 3.3.0
> Native build fails on macos due to getgrouplist not found
> -
>
> Key: YARN-8622
> URL: https://issues.apache.org/jira/browse/YARN-8622
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
>Affects Versions: 3.2.0, 3.3.0
> Environment: Darwin 17.7.0 Darwin Kernel Version 17.7.0: Thu Jun 21
> 22:53:14 PDT 2018; root:xnu-4570.71.2~1/RELEASE_X86_64 x86_64
> Apple LLVM version 9.1.0 (clang-902.0.39.2)
>Reporter: Ewan Higgs
>Assignee: Siyao Meng
>Priority: Major
> Attachments: YARN-8622.001.patch
>
>
> Usage of getgrouplist() is added in YARN-7221 and should affect Hadoop 3.2.0
> and later.
> Compiler:
> {code}
> $ /Library/Developer/CommandLineTools/usr/bin/c++ --version
> Apple LLVM version 9.1.0 (clang-902.0.39.2)
> Target: x86_64-apple-darwin17.7.0
> Thread model: posix
> InstalledDir: /Library/Developer/CommandLineTools/usr/bin
> {code}
> Build line:
> {code}
> [WARNING] /Library/Developer/CommandLineTools/usr/bin/c++ -g -O2 -Wall
> -pthread -D_FILE_OFFSET_BITS=64 -Wl,-search_paths_first
> -Wl,-headerpad_max_install_names
> CMakeFiles/test-oom-listener.dir/main/native/oom-listener/impl/oom_listener.c.o
>
> CMakeFiles/test-oom-listener.dir/main/native/oom-listener/test/oom_listener_test_main.cc.o
> -o test/test-oom-listener libgtest.a -lrt
> {code}
> Error message:
> {code}
> ...
> [WARNING]
> /Users/ehiggs/src/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/utils/docker-util.c:1264:12:
> error: no matching function for call to 'getgrouplist'
> [WARNING] int rc = getgrouplist(user, pw->pw_gid, groups, &ngroups);
> [WARNING]^~~~
> [WARNING] /usr/include/unistd.h:653:6: note: candidate function not viable:
> no known conversion from 'gid_t *' (aka 'unsigned int *') to 'int *' for 3rd
> argument
> [WARNING] int getgrouplist(const char *, int, int *, int *);
> [WARNING] ^
> [WARNING] In file included from
> /Users/ehiggs/src/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/test/utils/test_docker_util.cc:24:
> [WARNING]
> /Users/ehiggs/src/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/utils/docker-util.c:1271:9:
> error: no matching function for call to 'getgrouplist'
> [WARNING] if (getgrouplist(user, pw->pw_gid, groups, &ngroups) == -1) {
> [WARNING] ^~~~
> [WARNING] /usr/include/unistd.h:653:6: note: candidate function not viable:
> no known conversion from 'gid_t *' (aka 'unsigned int *') to 'int *' for 3rd
> argument
> [WARNING] int getgrouplist(const char *, int, int *, int *);
> [WARNING] ^
> [WARNING] 2 warnings and 2 errors generated.
> [WARNING] make[2]: ***
> [CMakeFiles/cetest.dir/main/native/container-executor/test/utils/test_docker_util.cc.o]
> Error 1
> [WARNING] make[1]: *** [CMakeFiles/cetest.dir/all] Error 2
> [WARNING] make: *** [all] Error 2
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8622) Native build fails on macos due to getgrouplist not found
[
https://issues.apache.org/jira/browse/YARN-8622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated YARN-8622:
-
Target Version/s: 3.3.0, 3.2.1
> Native build fails on macos due to getgrouplist not found
> -
>
> Key: YARN-8622
> URL: https://issues.apache.org/jira/browse/YARN-8622
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
>Affects Versions: 3.2.0, 3.3.0
> Environment: Darwin 17.7.0 Darwin Kernel Version 17.7.0: Thu Jun 21
> 22:53:14 PDT 2018; root:xnu-4570.71.2~1/RELEASE_X86_64 x86_64
> Apple LLVM version 9.1.0 (clang-902.0.39.2)
>Reporter: Ewan Higgs
>Assignee: Siyao Meng
>Priority: Major
> Attachments: YARN-8622.001.patch
>
>
> Usage of getgrouplist() is added in YARN-7221 and should affect Hadoop 3.2.0
> and later.
> Compiler:
> {code}
> $ /Library/Developer/CommandLineTools/usr/bin/c++ --version
> Apple LLVM version 9.1.0 (clang-902.0.39.2)
> Target: x86_64-apple-darwin17.7.0
> Thread model: posix
> InstalledDir: /Library/Developer/CommandLineTools/usr/bin
> {code}
> Build line:
> {code}
> [WARNING] /Library/Developer/CommandLineTools/usr/bin/c++ -g -O2 -Wall
> -pthread -D_FILE_OFFSET_BITS=64 -Wl,-search_paths_first
> -Wl,-headerpad_max_install_names
> CMakeFiles/test-oom-listener.dir/main/native/oom-listener/impl/oom_listener.c.o
>
> CMakeFiles/test-oom-listener.dir/main/native/oom-listener/test/oom_listener_test_main.cc.o
> -o test/test-oom-listener libgtest.a -lrt
> {code}
> Error message:
> {code}
> ...
> [WARNING]
> /Users/ehiggs/src/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/utils/docker-util.c:1264:12:
> error: no matching function for call to 'getgrouplist'
> [WARNING] int rc = getgrouplist(user, pw->pw_gid, groups, &ngroups);
> [WARNING]^~~~
> [WARNING] /usr/include/unistd.h:653:6: note: candidate function not viable:
> no known conversion from 'gid_t *' (aka 'unsigned int *') to 'int *' for 3rd
> argument
> [WARNING] int getgrouplist(const char *, int, int *, int *);
> [WARNING] ^
> [WARNING] In file included from
> /Users/ehiggs/src/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/test/utils/test_docker_util.cc:24:
> [WARNING]
> /Users/ehiggs/src/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/utils/docker-util.c:1271:9:
> error: no matching function for call to 'getgrouplist'
> [WARNING] if (getgrouplist(user, pw->pw_gid, groups, &ngroups) == -1) {
> [WARNING] ^~~~
> [WARNING] /usr/include/unistd.h:653:6: note: candidate function not viable:
> no known conversion from 'gid_t *' (aka 'unsigned int *') to 'int *' for 3rd
> argument
> [WARNING] int getgrouplist(const char *, int, int *, int *);
> [WARNING] ^
> [WARNING] 2 warnings and 2 errors generated.
> [WARNING] make[2]: ***
> [CMakeFiles/cetest.dir/main/native/container-executor/test/utils/test_docker_util.cc.o]
> Error 1
> [WARNING] make[1]: *** [CMakeFiles/cetest.dir/all] Error 2
> [WARNING] make: *** [all] Error 2
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9475) Create basic VE plugin
[
https://issues.apache.org/jira/browse/YARN-9475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16819222#comment-16819222
]
Hadoop QA commented on YARN-9475:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
15s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
59s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
34s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 42s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
27s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 17s{color} | {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager:
The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 13s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 20m
57s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
25s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 67m 26s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:bdbca0e |
| JIRA Issue | YARN-9475 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12966101/YARN-9475-004.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 48f684618524 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24
10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / bfcb653 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/23971/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/23971/testReport/ |
| Max. process+thread count | 44
[jira] [Commented] (YARN-9487) NodeManager shouldn't link against librt on macOS
[
https://issues.apache.org/jira/browse/YARN-9487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16819224#comment-16819224
]
Siyao Meng commented on YARN-9487:
--
And here's my short instruction for reviewers to test trunk does compile on
macOS with YARN-9487 (this one) and YARN-8622
{code:bash}
brew install maven openssl
git clone https://github.com/apache/hadoop.git trunk && cd trunk
wget
https://issues.apache.org/jira/secure/attachment/12966038/YARN-9487.001.patch
wget
https://issues.apache.org/jira/secure/attachment/12966111/YARN-8622.001.patch
git apply YARN-9487.001.patch
git apply YARN-8622.001.patch
mvn package -Pdist,native -DskipTests -e -Dmaven.javadoc.skip=true
-Denforcer.skip=true -DskipShade
-Dopenssl.lib=/usr/local/opt/openssl/lib/libcrypto.dylib
-Dopenssl.include=/usr/local/opt/openssl/include
{code}
> NodeManager shouldn't link against librt on macOS
> -
>
> Key: YARN-9487
> URL: https://issues.apache.org/jira/browse/YARN-9487
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
>Affects Versions: 3.2.0, 3.3.0
>Reporter: Siyao Meng
>Assignee: Siyao Meng
>Priority: Major
> Attachments: YARN-9487.001.patch
>
>
> I'm compiling hadoop/trunk native on a Mac. Found that when compiling
> NodeManager it is trying to link against librt on Darwin as well - and it
> failed for sure. Diff for a quick fix:
> {code}
> diff --git
> a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/CMakeLists.txt
>
> b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/CMakeLists.txt
> index f0f005d53b5..9c3f0135aaf 100644
> ---
> a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/CMakeLists.txt
> +++
> b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/CMakeLists.txt
> @@ -190,5 +190,10 @@ add_executable(test-oom-listener
> main/native/oom-listener/impl/oom_listener.h
> main/native/oom-listener/test/oom_listener_test_main.cc
> )
> -target_link_libraries(test-oom-listener gtest rt)
> +if(CMAKE_SYSTEM_NAME MATCHES "Darwin")
> +# macOS does not have rt
> +target_link_libraries(test-oom-listener gtest)
> +else()
> +target_link_libraries(test-oom-listener gtest rt)
> +endif()
> output_directory(test-oom-listener test)
> {code}
> This line of code is added in YARN-8437, so this affects 3.2.0+.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9487) NodeManager shouldn't link against librt on macOS
[
https://issues.apache.org/jira/browse/YARN-9487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated YARN-9487:
-
Description:
I'm compiling hadoop/trunk native on a Mac. Found that when compiling
NodeManager it is trying to link against librt on Darwin as well - and it
failed for sure. Diff for a quick fix:
{code}
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/CMakeLists.txt
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/CMakeLists.txt
index f0f005d53b5..9c3f0135aaf 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/CMakeLists.txt
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/CMakeLists.txt
@@ -190,5 +190,10 @@ add_executable(test-oom-listener
main/native/oom-listener/impl/oom_listener.h
main/native/oom-listener/test/oom_listener_test_main.cc
)
-target_link_libraries(test-oom-listener gtest rt)
+if(CMAKE_SYSTEM_NAME MATCHES "Darwin")
+# macOS does not have rt
+target_link_libraries(test-oom-listener gtest)
+else()
+target_link_libraries(test-oom-listener gtest rt)
+endif()
output_directory(test-oom-listener test)
{code}
This line of code is added in YARN-8437, so this affects 3.2.0+.
was:
I'm compiling hadoop/trunk native on a Mac. Found that when compiling
NodeManager it is trying to link against librt on Darwin as well - and it
failed for sure. Diff for a quick fix:
{code}
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/CMakeLists.txt
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/CMakeLists.txt
index f0f005d53b5..9c3f0135aaf 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/CMakeLists.txt
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/CMakeLists.txt
@@ -190,5 +190,10 @@ add_executable(test-oom-listener
main/native/oom-listener/impl/oom_listener.h
main/native/oom-listener/test/oom_listener_test_main.cc
)
-target_link_libraries(test-oom-listener gtest rt)
+if(CMAKE_SYSTEM_NAME MATCHES "Darwin")
+# macOS does not have rt
+target_link_libraries(test-oom-listener gtest)
+else()
+target_link_libraries(test-oom-listener gtest rt)
+endif()
output_directory(test-oom-listener test)
{code}
This line of code is added in YARN-8437, so this affects 3.2.0+.
Will upload the patch after I'm added to YARN contributor list.
> NodeManager shouldn't link against librt on macOS
> -
>
> Key: YARN-9487
> URL: https://issues.apache.org/jira/browse/YARN-9487
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
>Affects Versions: 3.2.0, 3.3.0
>Reporter: Siyao Meng
>Assignee: Siyao Meng
>Priority: Major
> Attachments: YARN-9487.001.patch
>
>
> I'm compiling hadoop/trunk native on a Mac. Found that when compiling
> NodeManager it is trying to link against librt on Darwin as well - and it
> failed for sure. Diff for a quick fix:
> {code}
> diff --git
> a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/CMakeLists.txt
>
> b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/CMakeLists.txt
> index f0f005d53b5..9c3f0135aaf 100644
> ---
> a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/CMakeLists.txt
> +++
> b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/CMakeLists.txt
> @@ -190,5 +190,10 @@ add_executable(test-oom-listener
> main/native/oom-listener/impl/oom_listener.h
> main/native/oom-listener/test/oom_listener_test_main.cc
> )
> -target_link_libraries(test-oom-listener gtest rt)
> +if(CMAKE_SYSTEM_NAME MATCHES "Darwin")
> +# macOS does not have rt
> +target_link_libraries(test-oom-listener gtest)
> +else()
> +target_link_libraries(test-oom-listener gtest rt)
> +endif()
> output_directory(test-oom-listener test)
> {code}
> This line of code is added in YARN-8437, so this affects 3.2.0+.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-9487) NodeManager shouldn't link against librt on macOS
[
https://issues.apache.org/jira/browse/YARN-9487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng reassigned YARN-9487:
Assignee: Siyao Meng
> NodeManager shouldn't link against librt on macOS
> -
>
> Key: YARN-9487
> URL: https://issues.apache.org/jira/browse/YARN-9487
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
>Affects Versions: 3.2.0, 3.3.0
>Reporter: Siyao Meng
>Assignee: Siyao Meng
>Priority: Major
> Attachments: YARN-9487.001.patch
>
>
> I'm compiling hadoop/trunk native on a Mac. Found that when compiling
> NodeManager it is trying to link against librt on Darwin as well - and it
> failed for sure. Diff for a quick fix:
> {code}
> diff --git
> a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/CMakeLists.txt
>
> b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/CMakeLists.txt
> index f0f005d53b5..9c3f0135aaf 100644
> ---
> a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/CMakeLists.txt
> +++
> b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/CMakeLists.txt
> @@ -190,5 +190,10 @@ add_executable(test-oom-listener
> main/native/oom-listener/impl/oom_listener.h
> main/native/oom-listener/test/oom_listener_test_main.cc
> )
> -target_link_libraries(test-oom-listener gtest rt)
> +if(CMAKE_SYSTEM_NAME MATCHES "Darwin")
> +# macOS does not have rt
> +target_link_libraries(test-oom-listener gtest)
> +else()
> +target_link_libraries(test-oom-listener gtest rt)
> +endif()
> output_directory(test-oom-listener test)
> {code}
> This line of code is added in YARN-8437, so this affects 3.2.0+.
> Will upload the patch after I'm added to YARN contributor list.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-8622) Native build fails on macos due to getgrouplist not found
[
https://issues.apache.org/jira/browse/YARN-8622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng reassigned YARN-8622:
Assignee: Siyao Meng
> Native build fails on macos due to getgrouplist not found
> -
>
> Key: YARN-8622
> URL: https://issues.apache.org/jira/browse/YARN-8622
> Project: Hadoop YARN
> Issue Type: Bug
> Environment: Darwin 17.7.0 Darwin Kernel Version 17.7.0: Thu Jun 21
> 22:53:14 PDT 2018; root:xnu-4570.71.2~1/RELEASE_X86_64 x86_64
> Apple LLVM version 9.1.0 (clang-902.0.39.2)
>Reporter: Ewan Higgs
>Assignee: Siyao Meng
>Priority: Major
>
> Compiler:
> {code}
> $ /Library/Developer/CommandLineTools/usr/bin/c++ --version
> Apple LLVM version 9.1.0 (clang-902.0.39.2)
> Target: x86_64-apple-darwin17.7.0
> Thread model: posix
> InstalledDir: /Library/Developer/CommandLineTools/usr/bin
> {code}
> Build line:
> {code}
> [WARNING] /Library/Developer/CommandLineTools/usr/bin/c++ -g -O2 -Wall
> -pthread -D_FILE_OFFSET_BITS=64 -Wl,-search_paths_first
> -Wl,-headerpad_max_install_names
> CMakeFiles/test-oom-listener.dir/main/native/oom-listener/impl/oom_listener.c.o
>
> CMakeFiles/test-oom-listener.dir/main/native/oom-listener/test/oom_listener_test_main.cc.o
> -o test/test-oom-listener libgtest.a -lrt
> {code}
> Error message:
> {code}
> ...
> [WARNING]
> /Users/ehiggs/src/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/utils/docker-util.c:1264:12:
> error: no matching function for call to 'getgrouplist'
> [WARNING] int rc = getgrouplist(user, pw->pw_gid, groups, &ngroups);
> [WARNING]^~~~
> [WARNING] /usr/include/unistd.h:653:6: note: candidate function not viable:
> no known conversion from 'gid_t *' (aka 'unsigned int *') to 'int *' for 3rd
> argument
> [WARNING] int getgrouplist(const char *, int, int *, int *);
> [WARNING] ^
> [WARNING] In file included from
> /Users/ehiggs/src/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/test/utils/test_docker_util.cc:24:
> [WARNING]
> /Users/ehiggs/src/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/utils/docker-util.c:1271:9:
> error: no matching function for call to 'getgrouplist'
> [WARNING] if (getgrouplist(user, pw->pw_gid, groups, &ngroups) == -1) {
> [WARNING] ^~~~
> [WARNING] /usr/include/unistd.h:653:6: note: candidate function not viable:
> no known conversion from 'gid_t *' (aka 'unsigned int *') to 'int *' for 3rd
> argument
> [WARNING] int getgrouplist(const char *, int, int *, int *);
> [WARNING] ^
> [WARNING] 2 warnings and 2 errors generated.
> [WARNING] make[2]: ***
> [CMakeFiles/cetest.dir/main/native/container-executor/test/utils/test_docker_util.cc.o]
> Error 1
> [WARNING] make[1]: *** [CMakeFiles/cetest.dir/all] Error 2
> [WARNING] make: *** [all] Error 2
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9475) Create basic VE plugin
[
https://issues.apache.org/jira/browse/YARN-9475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16819129#comment-16819129
]
Peter Bacsko commented on YARN-9475:
_Btw, I can see that this.binaryPath is modified many times in the constructor:
Does the ordering of the code represents some kind of precendence on the env
variables?_
Yes:
1. We try the explicit path, if given
2. Then we try a specific folder under {{$HADOOP_COMMON_HOME}}
3. Then go through some hard-wired directories
> Create basic VE plugin
> --
>
> Key: YARN-9475
> URL: https://issues.apache.org/jira/browse/YARN-9475
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
> Attachments: YARN-9475-001.patch, YARN-9475-002.patch,
> YARN-9475-003.patch, YARN-9475-004.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9475) Create basic VE plugin
[ https://issues.apache.org/jira/browse/YARN-9475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Bacsko updated YARN-9475: --- Attachment: YARN-9475-004.patch > Create basic VE plugin > -- > > Key: YARN-9475 > URL: https://issues.apache.org/jira/browse/YARN-9475 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Reporter: Peter Bacsko >Assignee: Peter Bacsko >Priority: Major > Attachments: YARN-9475-001.patch, YARN-9475-002.patch, > YARN-9475-003.patch, YARN-9475-004.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9123) Clean up and split testcases in TestNMWebServices for GPU support
[
https://issues.apache.org/jira/browse/YARN-9123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16819085#comment-16819085
]
Hadoop QA commented on YARN-9123:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
15s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
5s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 30s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
59s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
29s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
19s{color} | {color:green}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager:
The patch generated 0 new + 5 unchanged - 1 fixed = 5 total (was 6) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 51s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 21m
0s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
27s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 70m 33s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:bdbca0e |
| JIRA Issue | YARN-9123 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12966087/YARN-9123.007.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux cb97edcf5ca0 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2
17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 008766c |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/23970/testReport/ |
| Max. process+thread count | 447 (vs. ulimit of 1) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/23970/console |
| Powered by | Apache
[jira] [Commented] (YARN-9440) Improve diagnostics for scheduler and app activities
[ https://issues.apache.org/jira/browse/YARN-9440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16819081#comment-16819081 ] Weiwei Yang commented on YARN-9440: --- Thanks [~Tao Yang], I'll help to review the patch this week. > Improve diagnostics for scheduler and app activities > > > Key: YARN-9440 > URL: https://issues.apache.org/jira/browse/YARN-9440 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacityscheduler >Reporter: Tao Yang >Assignee: Tao Yang >Priority: Major > Attachments: YARN-9440.001.patch > > > [Design > doc|https://docs.google.com/document/d/1pwf-n3BCLW76bGrmNPM4T6pQ3vC4dVMcN2Ud1hq1t2M/edit#heading=h.cyw6zeehzqmx] > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9475) Create basic VE plugin
[ https://issues.apache.org/jira/browse/YARN-9475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16819051#comment-16819051 ] Peter Bacsko commented on YARN-9475: Thanks [~snemeth] very reasonable comments. I'm working on the items. > Create basic VE plugin > -- > > Key: YARN-9475 > URL: https://issues.apache.org/jira/browse/YARN-9475 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Reporter: Peter Bacsko >Assignee: Peter Bacsko >Priority: Major > Attachments: YARN-9475-001.patch, YARN-9475-002.patch, > YARN-9475-003.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9482) DistributedShell job with localization fails in unsecure cluster
[
https://issues.apache.org/jira/browse/YARN-9482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16819038#comment-16819038
]
Hadoop QA commented on YARN-9482:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
20s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m
59s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 17s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
20s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 14s{color} | {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell:
The patch generated 1 new + 206 unchanged - 0 fixed = 207 total (was 206)
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 41s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 20m 36s{color}
| {color:red} hadoop-yarn-applications-distributedshell in the patch failed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
29s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 72m 12s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.applications.distributedshell.TestDistributedShell |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:bdbca0e |
| JIRA Issue | YARN-9482 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12966070/YARN-9482-001.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux b07b91d8406d 4.4.0-144-generic #170~14.04.1-Ubuntu SMP Mon Mar
18 15:02:05 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 008766c |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/23969/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-applications_hadoop-yarn-applications-distributedshell.txt
|
| unit |
https://builds.apache.org/job/PreCommit-YARN-Build/23969/artifact/out/p
[jira] [Updated] (YARN-9491) TestApplicationMasterServiceFair>ApplicationMasterServiceTestBase.testUpdateTrackingUrl fails intermittent
[
https://issues.apache.org/jira/browse/YARN-9491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prabhu Joseph updated YARN-9491:
Affects Version/s: 3.2.0
> TestApplicationMasterServiceFair>ApplicationMasterServiceTestBase.testUpdateTrackingUrl
> fails intermittent
> --
>
> Key: YARN-9491
> URL: https://issues.apache.org/jira/browse/YARN-9491
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 3.2.0
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Minor
>
> TestApplicationMasterServiceFair>ApplicationMasterServiceTestBase.testUpdateTrackingUrl
> fails intermittent.
> {code}
> Error Message
> expected:<[hadoop.apache.org]> but was:<[N/A]>
> Stacktrace
> org.junit.ComparisonFailure: expected:<[hadoop.apache.org]> but was:<[N/A]>
> at org.junit.Assert.assertEquals(Assert.java:115)
> at org.junit.Assert.assertEquals(Assert.java:144)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterServiceTestBase.testUpdateTrackingUrl(ApplicationMasterServiceTestBase.java:467)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.lang.Thread.run(Thread.java:748)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-9492) TestRMEmbeddedElector.testCallbackSynchronization fails intermittent
Prabhu Joseph created YARN-9492:
---
Summary: TestRMEmbeddedElector.testCallbackSynchronization fails
intermittent
Key: YARN-9492
URL: https://issues.apache.org/jira/browse/YARN-9492
Project: Hadoop YARN
Issue Type: Bug
Components: resourcemanager
Affects Versions: 3.2.0
Reporter: Prabhu Joseph
Assignee: Prabhu Joseph
TestRMEmbeddedElector.testCallbackSynchronization fails intermittent
{code}
Error Message
org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode =
ConnectionLoss
Stacktrace
org.apache.hadoop.service.ServiceStateException:
org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode =
ConnectionLoss
at
org.apache.hadoop.service.ServiceStateException.convert(ServiceStateException.java:105)
at
org.apache.hadoop.service.AbstractService.init(AbstractService.java:173)
at
org.apache.hadoop.yarn.server.resourcemanager.TestRMEmbeddedElector.testCallbackSynchronization(TestRMEmbeddedElector.java:156)
at
org.apache.hadoop.yarn.server.resourcemanager.TestRMEmbeddedElector.testCallbackSynchronization(TestRMEmbeddedElector.java:117)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.apache.zookeeper.JUnit4ZKTestRunner$LoggedInvokeMethod.evaluate(JUnit4ZKTestRunner.java:55)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.rules.TestWatchman$1.evaluate(TestWatchman.java:53)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
at
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
at
org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
at
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
at
org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
at
org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException:
KeeperErrorCode = ConnectionLoss
at org.apache.zookeeper.KeeperException.create(KeeperException.java:102)
at
org.apache.hadoop.ha.ActiveStandbyElector$WatcherWithClientRef.waitForZKConnectionEvent(ActiveStandbyElector.java:1165)
at
org.apache.hadoop.ha.ActiveStandbyElector$WatcherWithClientRef.access$400(ActiveStandbyElector.java:1136)
at
org.apache.hadoop.ha.ActiveStandbyElector.connectToZooKeeper(ActiveStandbyElector.java:699)
at
org.apache.hadoop.ha.ActiveStandbyElector.createConnection(ActiveStandbyElector.java:853)
at
org.apache.hadoop.ha.ActiveStandbyElector.ensureParentZNode(ActiveStandbyElector.java:336)
at
org.apache.hadoop.yarn.server.resourcemanager.ActiveStandbyElectorBasedElectorService.serviceInit(ActiveStandbyElectorBasedElectorService.java:110)
at
org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
... 31 more
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-
[jira] [Commented] (YARN-9123) Clean up and split testcases in TestNMWebServices for GPU support
[ https://issues.apache.org/jira/browse/YARN-9123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16819015#comment-16819015 ] Szilard Nemeth commented on YARN-9123: -- Hi [~jojochuang]! Please see the latest patch, this should fix the checkstyle issue! Thanks! > Clean up and split testcases in TestNMWebServices for GPU support > - > > Key: YARN-9123 > URL: https://issues.apache.org/jira/browse/YARN-9123 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Szilard Nemeth >Assignee: Szilard Nemeth >Priority: Minor > Attachments: YARN-9123.001.patch, YARN-9123.002.patch, > YARN-9123.003.patch, YARN-9123.004.patch, YARN-9123.005.patch, > YARN-9123.006.patch, YARN-9123.007.patch > > > The following testcases can be cleaned up a bit: > TestNMWebServices#testGetNMResourceInfo - Can be split up to 3 different cases > TestNMWebServices#testGetYarnGpuResourceInfo -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9123) Clean up and split testcases in TestNMWebServices for GPU support
[ https://issues.apache.org/jira/browse/YARN-9123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szilard Nemeth updated YARN-9123: - Attachment: YARN-9123.007.patch > Clean up and split testcases in TestNMWebServices for GPU support > - > > Key: YARN-9123 > URL: https://issues.apache.org/jira/browse/YARN-9123 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Szilard Nemeth >Assignee: Szilard Nemeth >Priority: Minor > Attachments: YARN-9123.001.patch, YARN-9123.002.patch, > YARN-9123.003.patch, YARN-9123.004.patch, YARN-9123.005.patch, > YARN-9123.006.patch, YARN-9123.007.patch > > > The following testcases can be cleaned up a bit: > TestNMWebServices#testGetNMResourceInfo - Can be split up to 3 different cases > TestNMWebServices#testGetYarnGpuResourceInfo -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9483) DistributedShell does not release container when failed to localize at launch
[
https://issues.apache.org/jira/browse/YARN-9483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16819006#comment-16819006
]
Prabhu Joseph commented on YARN-9483:
-
Failed testcase is not related and will be fixed by YARN-9452.
> DistributedShell does not release container when failed to localize at launch
> -
>
> Key: YARN-9483
> URL: https://issues.apache.org/jira/browse/YARN-9483
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Major
> Attachments: YARN-9483-001.patch
>
>
> DistributedShell does not release container when failed to localize at
> launch. The launch threads does not increment completed & failed containers
> when failed to localize. And the main thread waits for the containers to
> complete without failing the job.
> {code}
> yarn jar
> /HADOOP/hadoop-3.2.0/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-3.2.0.jar
> -shell_command ls -shell_args / -jar
> /HADOOP/hadoop-3.2.0/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-3.2.0.jar
> -localize_files /tmp/prabhu
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9470) Fix order of actual and expected expression in assert statements
[
https://issues.apache.org/jira/browse/YARN-9470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16818994#comment-16818994
]
Prabhu Joseph commented on YARN-9470:
-
Hi [~ajisakaa], Can you review this jira when you get time. This fixes
misplaced expected and actual args in {{assertEquals}} with AssertJ.
Failed testcases are not related - will be handled as part of YARN-9491 and
YARN-6272.
> Fix order of actual and expected expression in assert statements
>
>
> Key: YARN-9470
> URL: https://issues.apache.org/jira/browse/YARN-9470
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: yarn
>Affects Versions: 3.2.0
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Major
> Attachments: YARN-9470-001.patch, YARN-9470-002.patch,
> YARN-9470-003.patch, assertEquals
>
>
> Fix order of actual and expected expression in assert statements which gives
> misleading message when test case fails. Attached file has some of the places
> where it is placed wrongly.
> {code}
> [ERROR]
> testNodeRemovalGracefully(org.apache.hadoop.yarn.server.resourcemanager.TestResourceTrackerService)
> Time elapsed: 3.385 s <<< FAILURE!
> java.lang.AssertionError: Shutdown nodes should be 0 now expected:<1> but
> was:<0>
> {code}
> For long term, [AssertJ|http://joel-costigliola.github.io/assertj/] can be
> used for new test cases which avoids such mistakes.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-9491) TestApplicationMasterServiceFair>ApplicationMasterServiceTestBase.testUpdateTrackingUrl fails intermittent
Prabhu Joseph created YARN-9491:
---
Summary:
TestApplicationMasterServiceFair>ApplicationMasterServiceTestBase.testUpdateTrackingUrl
fails intermittent
Key: YARN-9491
URL: https://issues.apache.org/jira/browse/YARN-9491
Project: Hadoop YARN
Issue Type: Bug
Reporter: Prabhu Joseph
Assignee: Prabhu Joseph
TestApplicationMasterServiceFair>ApplicationMasterServiceTestBase.testUpdateTrackingUrl
fails intermittent.
{code}
Error Message
expected:<[hadoop.apache.org]> but was:<[N/A]>
Stacktrace
org.junit.ComparisonFailure: expected:<[hadoop.apache.org]> but was:<[N/A]>
at org.junit.Assert.assertEquals(Assert.java:115)
at org.junit.Assert.assertEquals(Assert.java:144)
at
org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterServiceTestBase.testUpdateTrackingUrl(ApplicationMasterServiceTestBase.java:467)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
at
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.lang.Thread.run(Thread.java:748)
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9488) Skip YARNFeatureNotEnabledException from ClientRMService
[
https://issues.apache.org/jira/browse/YARN-9488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16818972#comment-16818972
]
Hadoop QA commented on YARN-9488:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 31m
59s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 76m
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
34s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 26s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
15s{color} | {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
in trunk has 2 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
34s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 52s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 88m 32s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
36s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}231m 14s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.yarn.server.resourcemanager.TestRMEmbeddedElector
|
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:bdbca0e |
| JIRA Issue | YARN-9488 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12966049/YARN-9488-001.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 86ac456a4c99 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2
17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / a5ceed2 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| findbugs | v3.1.0-RC1 |
| findbugs |
https://builds.apache.org/job/PreCommit-YARN-Build/23967/artifact/out/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-warnings.html
|
| unit |
https://builds.apache.org/j
[jira] [Commented] (YARN-9123) Clean up and split testcases in TestNMWebServices for GPU support
[ https://issues.apache.org/jira/browse/YARN-9123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16818964#comment-16818964 ] Wei-Chiu Chuang commented on YARN-9123: --- Please update the patch to address the checkstyle warning. +1 after that. Thanks! > Clean up and split testcases in TestNMWebServices for GPU support > - > > Key: YARN-9123 > URL: https://issues.apache.org/jira/browse/YARN-9123 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Szilard Nemeth >Assignee: Szilard Nemeth >Priority: Minor > Attachments: YARN-9123.001.patch, YARN-9123.002.patch, > YARN-9123.003.patch, YARN-9123.004.patch, YARN-9123.005.patch, > YARN-9123.006.patch > > > The following testcases can be cleaned up a bit: > TestNMWebServices#testGetNMResourceInfo - Can be split up to 3 different cases > TestNMWebServices#testGetYarnGpuResourceInfo -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9482) DistributedShell job with localization fails in unsecure cluster
[
https://issues.apache.org/jira/browse/YARN-9482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prabhu Joseph updated YARN-9482:
Attachment: YARN-9482-001.patch
> DistributedShell job with localization fails in unsecure cluster
>
>
> Key: YARN-9482
> URL: https://issues.apache.org/jira/browse/YARN-9482
> Project: Hadoop YARN
> Issue Type: Bug
> Components: distributed-shell
>Affects Versions: 3.3.0
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Major
> Attachments: YARN-9482-001.patch
>
>
> DistributedShell job with localization fails in unsecure cluster. The client
> localizes the input files to home directory (job user) whereas the AM runs as
> yarn user reads from it's home directory.
> *Command:*
> {code}
> yarn jar
> /HADOOP/hadoop-3.2.0/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-3.2.0.jar
> -shell_command ls -shell_args / -jar
> /HADOOP/hadoop-3.2.0/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-3.2.0.jar
> -localize_files /tmp/prabhu
> {code}
> {code}
> Exception in thread "Thread-4" java.io.UncheckedIOException: Error during
> localization setup
> at
> org.apache.hadoop.yarn.applications.distributedshell.ApplicationMaster$LaunchContainerRunnable.lambda$run$0(ApplicationMaster.java:1495)
> at
> java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1382)
> at
> java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:580)
> at
> org.apache.hadoop.yarn.applications.distributedshell.ApplicationMaster$LaunchContainerRunnable.run(ApplicationMaster.java:1481)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.io.FileNotFoundException: File does not exist:
> hdfs://yarn-ats-1:8020/user/yarn/DistributedShell/application_1554817981283_0003/prabhu
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1586)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1579)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1594)
> at
> org.apache.hadoop.yarn.applications.distributedshell.ApplicationMaster$LaunchContainerRunnable.lambda$run$0(ApplicationMaster.java:1487)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9490) applicationresourceusagereport return wrong number of reserved containers
[ https://issues.apache.org/jira/browse/YARN-9490?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] yanbing zhang updated YARN-9490: Attachment: YARN-9490.patch > applicationresourceusagereport return wrong number of reserved containers > - > > Key: YARN-9490 > URL: https://issues.apache.org/jira/browse/YARN-9490 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 3.3.0 >Reporter: yanbing zhang >Priority: Minor > Fix For: 3.3.0 > > Attachments: YARN-9490.patch > > > when getting an ApplicationResourceUsageReport instance from the class of > SchedulerApplicationAttempt, I found the input constructor > parameter(reservedContainers.size()) is wrong. because the type of this > variable is Map>, so > "reservedContainer.size()" is not the number of containers, but the number of > SchedulerRequestKey. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-9490) applicationresourceusagereport return wrong number of reserved containers
yanbing zhang created YARN-9490: --- Summary: applicationresourceusagereport return wrong number of reserved containers Key: YARN-9490 URL: https://issues.apache.org/jira/browse/YARN-9490 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager Affects Versions: 3.3.0 Reporter: yanbing zhang Fix For: 3.3.0 when getting an ApplicationResourceUsageReport instance from the class of SchedulerApplicationAttempt, I found the input constructor parameter(reservedContainers.size()) is wrong. because the type of this variable is Map>, so "reservedContainer.size()" is not the number of containers, but the number of SchedulerRequestKey. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9475) Create basic VE plugin
[
https://issues.apache.org/jira/browse/YARN-9475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16818945#comment-16818945
]
Szilard Nemeth commented on YARN-9475:
--
Thanks [~pbacsko] for this patch!
Couple of comments:
1. The constructor
{{org.apache.hadoop.yarn.server.nodemanager.containermanager.resourceplugin.com.nec.NECVEPlugin#NECVEPlugin(java.util.function.Function,
java.lang.String[])}} could be private, but as you said offline, you want to
use this from tests later, so this is a no-op.
2. IntelliJ complains about the {{binaryName}} / {{binaryFile}} fields: They
could be local variables as you only use them from inside the constructor. Do
you really need these as fields?
3. In the constructor, there's this:
{code:java}
if (null != envBinaryName) {
{code}
I think the null check is more readable on the other way around.
4. {{LOG.warn("Specified path is a directory, falling back");}} --> Log could
be improved to describe what we are falling back to.
5. Please extract methods, the constructor is 50 lines long, this is too much.
I can see at least 3 methods here:
- A method that reads the binary path based on the
{{NECVEPlugin#ENV_SCRIPT_PATH}}
- A method that reads the binary path based on the
{{NECVEPlugin#HADOOP_COMMON_HOME}}
- A method that reads {{binaryFile}} / {{binaryPath}} based on the
{{scriptPaths}} variable.
Btw, I can see that this.binaryPath is modified many times in the constructor:
Does the ordering of the code represents some kind of precendence on the env
variables?
6. In {{NECVEPlugin#getDevices}}: I can see no reason to declare the output
String outside of the if. Please declare + assign in one go inside the
try-catch block!
7. {{NECVEPlugin#onDevicesAllocated}} does not throw any Exception so the
signature should not contains a throws-clause.
8. In {{NECVEPlugin#parseOutput}}: keyvalues should be keyValues instead (note
the uppercase 'V').
9. In {{NECVEPlugin#parseOutput}}: The declaration of device could be moved to
the assignment.
10. In {{NECVEPlugin#parseOutput}}:
{{LOG.error("Unknown format of script output! Skip this line");}} --> Text
should start with "Skipping" instead.
11. In {{NECVEPlugin#parseOutput}}: There's a typo in this comment:
{code:java}
// for key value pars{code}
12. In {{NECVEPlugin#parseOutput}}: The long if-chain should be if-else for
every key, right? Assuming that for one key, we should not update the builder
twice.
Moreover, I would prefer to use a Map where you store keys as strings (like
"id", "dev", etc.) and Function object as values, that can act on the builder,
like invoking a set operation on it.
This way, the whole if-chain could be eliminated.
13. In {{NECVEPlugin#allocateDevices}}: The Device type parameter is not
required for the {{HashSet}}.
> Create basic VE plugin
> --
>
> Key: YARN-9475
> URL: https://issues.apache.org/jira/browse/YARN-9475
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
> Attachments: YARN-9475-001.patch, YARN-9475-002.patch,
> YARN-9475-003.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6929) yarn.nodemanager.remote-app-log-dir structure is not scalable
[
https://issues.apache.org/jira/browse/YARN-6929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16818918#comment-16818918
]
Peter Bacsko commented on YARN-6929:
Latest patch LGTM +1 (non-binding)
> yarn.nodemanager.remote-app-log-dir structure is not scalable
> -
>
> Key: YARN-6929
> URL: https://issues.apache.org/jira/browse/YARN-6929
> Project: Hadoop YARN
> Issue Type: Bug
> Components: log-aggregation
>Affects Versions: 2.7.3
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Major
> Attachments: YARN-6929-007.patch, YARN-6929-008.patch,
> YARN-6929-009.patch, YARN-6929-010.patch, YARN-6929.1.patch,
> YARN-6929.2.patch, YARN-6929.2.patch, YARN-6929.3.patch, YARN-6929.4.patch,
> YARN-6929.5.patch, YARN-6929.6.patch, YARN-6929.patch
>
>
> The current directory structure for yarn.nodemanager.remote-app-log-dir is
> not scalable. Maximum Subdirectory limit by default is 1048576 (HDFS-6102).
> With retention yarn.log-aggregation.retain-seconds of 7days, there are more
> chances LogAggregationService fails to create a new directory with
> FSLimitException$MaxDirectoryItemsExceededException.
> The current structure is
> //logs/. This can be
> improved with adding date as a subdirectory like
> //logs//
> {code}
> WARN
> org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService:
> Application failed to init aggregation
> org.apache.hadoop.yarn.exceptions.YarnRuntimeException:
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.protocol.FSLimitException$MaxDirectoryItemsExceededException):
> The directory item limit of /app-logs/yarn/logs is exceeded: limit=1048576
> items=1048576
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.verifyMaxDirItems(FSDirectory.java:2021)
>
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.addChild(FSDirectory.java:2072)
>
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.unprotectedMkdir(FSDirectory.java:1841)
>
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirsRecursively(FSNamesystem.java:4351)
>
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirsInternal(FSNamesystem.java:4262)
>
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirsInt(FSNamesystem.java:4221)
>
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirs(FSNamesystem.java:4194)
>
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.mkdirs(NameNodeRpcServer.java:813)
>
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.mkdirs(ClientNamenodeProtocolServerSideTranslatorPB.java:600)
>
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
>
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:619)
>
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:962)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2039)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2035)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
>
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2033)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService.createAppDir(LogAggregationService.java:308)
>
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService.initAppAggregator(LogAggregationService.java:366)
>
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService.initApp(LogAggregationService.java:320)
>
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService.handle(LogAggregationService.java:443)
>
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService.handle(LogAggregationService.java:67)
>
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:173)
>
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:106)
> at java.lang.Thread.run(Thread.java:745)
> Caused by:
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.protocol.FSLimitException$MaxDirectoryItemsExceededException):
> The directory item limit of /app-logs/yarn/logs is exceeded: limit=1048576
> items=1048576
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.verifyMaxDirItems(FSDirectory.java:2021)
>
> at
> org.apache.hadoop.hdfs.server.na
[jira] [Created] (YARN-9489) Support filtering by request-priorities and allocation-request-ids for query results of app activities
Tao Yang created YARN-9489: -- Summary: Support filtering by request-priorities and allocation-request-ids for query results of app activities Key: YARN-9489 URL: https://issues.apache.org/jira/browse/YARN-9489 Project: Hadoop YARN Issue Type: Sub-task Reporter: Tao Yang Assignee: Tao Yang [Design Doc #4.2|https://docs.google.com/document/d/1pwf-n3BCLW76bGrmNPM4T6pQ3vC4dVMcN2Ud1hq1t2M/edit#heading=h.m04tqsosk94h] -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9470) Fix order of actual and expected expression in assert statements
[
https://issues.apache.org/jira/browse/YARN-9470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16818864#comment-16818864
]
Hadoop QA commented on YARN-9470:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
18s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 64 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
2s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
16s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 8m
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
22m 10s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests
{color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
24s{color} | {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
in trunk has 2 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 6m
53s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
16s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
18s{color} | {color:green} hadoop-yarn-project/hadoop-yarn: The patch generated
0 new + 1937 unchanged - 13 fixed = 1937 total (was 1950) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 6m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
15s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 53s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests
{color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 10m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 5m
32s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
55s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
40s{color} | {color:green} hadoop-yarn-server-common in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 20m
57s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
59s{color} | {color:green} hadoop-yarn-
[jira] [Commented] (YARN-9483) DistributedShell does not release container when failed to localize at launch
[
https://issues.apache.org/jira/browse/YARN-9483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16818855#comment-16818855
]
Hadoop QA commented on YARN-9483:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
16s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 39s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
17s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 22s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 20m 50s{color}
| {color:red} hadoop-yarn-applications-distributedshell in the patch failed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
22s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 65m 35s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.applications.distributedshell.TestDistributedShell |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:bdbca0e |
| JIRA Issue | YARN-9483 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12966050/YARN-9483-001.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 2f0fe0fbcebf 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2
17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / a5ceed2 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-YARN-Build/23968/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-applications_hadoop-yarn-applications-distributedshell.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/23968/testReport/ |
| Max. process+thread count | 657 (vs. ulimit o
[jira] [Commented] (YARN-9080) Bucket Directories as part of ATS done accumulates
[ https://issues.apache.org/jira/browse/YARN-9080?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16818853#comment-16818853 ] Peter Bacsko commented on YARN-9080: +1 (non-binding) > Bucket Directories as part of ATS done accumulates > -- > > Key: YARN-9080 > URL: https://issues.apache.org/jira/browse/YARN-9080 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Prabhu Joseph >Assignee: Prabhu Joseph >Priority: Major > Attachments: 0001-YARN-9080.patch, 0002-YARN-9080.patch, > 0003-YARN-9080.patch, YARN-9080-004.patch, YARN-9080-005.patch, > YARN-9080-006.patch, YARN-9080-007.patch, YARN-9080-008.patch > > > Have observed older bucket directories cluster_timestamp, bucket1 and bucket2 > as part of ATS done accumulates. The cleanLogs part of EntityLogCleaner > removes only the app directories and not the bucket directories. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-9472) Add multi-thread asynchronous scheduling to fair scheduler
[ https://issues.apache.org/jira/browse/YARN-9472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16818695#comment-16818695 ] Tao Yang edited comment on YARN-9472 at 4/16/19 9:51 AM: - Hi, [~zhuqi]. Multi-thread asynchronous scheduling only means support scheduling in multiple threads, scheduling approaches can be classified as node-based and request-based, currently fair scheduler only support node-based and capacity scheduler can support both, global scheduler is request-based and aims for enhancing scheduling performance and considering complex resource placement requirements. I think your purpose is to support global scheduler including multi-thread asynchronous scheduling, multi-nodes lookup mechanism, placement constraints etc for fair scheduler, Right? was (Author: tao yang): Hi, [~zhuqi]. Multi-thread asynchronous scheduling only means support scheduling in multiple threads, scheduling approaches can be classified as node-based and request-based, currently fair scheduler only support node-based and capacity scheduler can support both, global scheduler is request-based and aims for enhance scheduling performance with considering complex resource placement requirements. I think your purpose is to support global scheduler including multi-thread asynchronous scheduling, multi-nodes lookup mechanism, placement constraints etc for fair scheduler, Right? > Add multi-thread asynchronous scheduling to fair scheduler > -- > > Key: YARN-9472 > URL: https://issues.apache.org/jira/browse/YARN-9472 > Project: Hadoop YARN > Issue Type: Sub-task > Components: fairscheduler, resourcemanager >Reporter: zhuqi >Priority: Major > > Now the capacity scheduler has multi-thread asynchronous scheduling, i think > the fair scheduler also need to support it. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9483) DistributedShell does not release container when failed to localize at launch
[
https://issues.apache.org/jira/browse/YARN-9483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prabhu Joseph updated YARN-9483:
Attachment: YARN-9483-001.patch
> DistributedShell does not release container when failed to localize at launch
> -
>
> Key: YARN-9483
> URL: https://issues.apache.org/jira/browse/YARN-9483
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Major
> Attachments: YARN-9483-001.patch
>
>
> DistributedShell does not release container when failed to localize at
> launch. The launch threads does not increment completed & failed containers
> when failed to localize. And the main thread waits for the containers to
> complete without failing the job.
> {code}
> yarn jar
> /HADOOP/hadoop-3.2.0/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-3.2.0.jar
> -shell_command ls -shell_args / -jar
> /HADOOP/hadoop-3.2.0/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-3.2.0.jar
> -localize_files /tmp/prabhu
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9488) Skip YARNFeatureNotEnabledException from ClientRMService
[
https://issues.apache.org/jira/browse/YARN-9488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prabhu Joseph updated YARN-9488:
Attachment: YARN-9488-001.patch
> Skip YARNFeatureNotEnabledException from ClientRMService
>
>
> Key: YARN-9488
> URL: https://issues.apache.org/jira/browse/YARN-9488
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: resourcemanager
>Affects Versions: 3.2.0
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Minor
> Attachments: YARN-9488-001.patch
>
>
> RM logs are accumulated with YARNFeatureNotEnabledException when running
> DIstributed Shell jobs while {{ClientRMService#getResourceProfiles}}
> {code}
> 2019-04-16 07:10:47,699 INFO org.apache.hadoop.ipc.Server: IPC Server handler
> 0 on 8050, call Call#5 Retry#0
> org.apache.hadoop.yarn.api.ApplicationClientProtocolPB.getResourceProfiles
> from 172.26.81.91:41198
> org.apache.hadoop.yarn.exceptions.YARNFeatureNotEnabledException: Resource
> profile is not enabled, please enable resource profile feature before using
> its functions. (by setting yarn.resourcemanager.resource-profiles.enabled to
> true)
> at
> org.apache.hadoop.yarn.server.resourcemanager.resource.ResourceProfilesManagerImpl.checkAndThrowExceptionWhenFeatureDisabled(ResourceProfilesManagerImpl.java:191)
> at
> org.apache.hadoop.yarn.server.resourcemanager.resource.ResourceProfilesManagerImpl.getResourceProfiles(ResourceProfilesManagerImpl.java:214)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.getResourceProfiles(ClientRMService.java:1833)
> at
> org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl.getResourceProfiles(ApplicationClientProtocolPBServiceImpl.java:670)
> at
> org.apache.hadoop.yarn.proto.ApplicationClientProtocol$ApplicationClientProtocolService$2.callBlockingMethod(ApplicationClientProtocol.java:665)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:524)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-9488) Skip YARNFeatureNotEnabledException from ClientRMService
Prabhu Joseph created YARN-9488:
---
Summary: Skip YARNFeatureNotEnabledException from ClientRMService
Key: YARN-9488
URL: https://issues.apache.org/jira/browse/YARN-9488
Project: Hadoop YARN
Issue Type: Improvement
Components: resourcemanager
Affects Versions: 3.2.0
Reporter: Prabhu Joseph
Assignee: Prabhu Joseph
RM logs are accumulated with YARNFeatureNotEnabledException when running
DIstributed Shell jobs while {{ClientRMService#getResourceProfiles}}
{code}
2019-04-16 07:10:47,699 INFO org.apache.hadoop.ipc.Server: IPC Server handler 0
on 8050, call Call#5 Retry#0
org.apache.hadoop.yarn.api.ApplicationClientProtocolPB.getResourceProfiles from
172.26.81.91:41198
org.apache.hadoop.yarn.exceptions.YARNFeatureNotEnabledException: Resource
profile is not enabled, please enable resource profile feature before using its
functions. (by setting yarn.resourcemanager.resource-profiles.enabled to true)
at
org.apache.hadoop.yarn.server.resourcemanager.resource.ResourceProfilesManagerImpl.checkAndThrowExceptionWhenFeatureDisabled(ResourceProfilesManagerImpl.java:191)
at
org.apache.hadoop.yarn.server.resourcemanager.resource.ResourceProfilesManagerImpl.getResourceProfiles(ResourceProfilesManagerImpl.java:214)
at
org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.getResourceProfiles(ClientRMService.java:1833)
at
org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl.getResourceProfiles(ApplicationClientProtocolPBServiceImpl.java:670)
at
org.apache.hadoop.yarn.proto.ApplicationClientProtocol$ApplicationClientProtocolService$2.callBlockingMethod(ApplicationClientProtocol.java:665)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:524)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682)
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9487) NodeManager shouldn't link against librt on macOS
[
https://issues.apache.org/jira/browse/YARN-9487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16818757#comment-16818757
]
Hadoop QA commented on YARN-9487:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
25s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 23m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
9s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
36m 33s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 26s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 22m
39s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
52s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 76m 44s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:bdbca0e |
| JIRA Issue | YARN-9487 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12966038/YARN-9487.001.patch |
| Optional Tests | dupname asflicense compile cc mvnsite javac unit |
| uname | Linux bb9225d57391 4.4.0-141-generic #167~14.04.1-Ubuntu SMP Mon Dec
10 13:20:24 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / a5ceed2 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/23966/testReport/ |
| Max. process+thread count | 340 (vs. ulimit of 1) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/23966/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> NodeManager shouldn't link against librt on macOS
> -
>
> Key: YARN-9487
> URL: https://issues.apache.org/jira/browse/YARN-9487
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
>Affects Versions: 3.2.0, 3.3.0
>Reporter: Siyao Meng
>Priority: Major
> Attachments: YARN-9487.001.patch
>
>
> I'm compiling hadoop/trunk native on a Mac. Found that when compiling
> NodeManager it is trying to link against librt on Darwin as well - and it
> failed for sure. Diff for a quick fix:
> {code}
>
[jira] [Commented] (YARN-9452) Timeline related testcases are failing
[
https://issues.apache.org/jira/browse/YARN-9452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16818746#comment-16818746
]
Prabhu Joseph commented on YARN-9452:
-
Hi [~giovanni.fumarola], Can you review this jira as well when you get some
time.This fixes failing timeline testcases. Thanks.
> Timeline related testcases are failing
> --
>
> Key: YARN-9452
> URL: https://issues.apache.org/jira/browse/YARN-9452
> Project: Hadoop YARN
> Issue Type: Bug
> Components: ATSv2, test
>Affects Versions: 3.2.0
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Major
> Attachments: YARN-9452-001.patch, YARN-9452-002.patch
>
>
> Timeline related testcases are failing.
> TestDistributedShell#testDSShellWithoutDomainV2CustomizedFlow
> {code}
> [ERROR]
> testDSShellWithoutDomainV2CustomizedFlow(org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell)
> Time elapsed: 72.14 s <<< FAILURE!
> java.lang.AssertionError: Entity ID prefix should be same across each publish
> of same entity expected:<9223372036854775806> but was:<9223370482298585580>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:834)
> at org.junit.Assert.assertEquals(Assert.java:645)
> at
> org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell.verifyEntityForTimelineV2(TestDistributedShell.java:695)
> at
> org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell.checkTimelineV2(TestDistributedShell.java:588)
> at
> org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell.testDSShell(TestDistributedShell.java:459)
> at
> org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell.testDSShellWithoutDomainV2CustomizedFlow(TestDistributedShell.java:330)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.lang.Thread.run(Thread.java:748)
> {code}
> TestTimelineAuthFilterForV2#testPutTimelineEntities
> {code}
> [ERROR]
> testPutTimelineEntities[3](org.apache.hadoop.yarn.server.timelineservice.security.TestTimelineAuthFilterForV2)
> Time elapsed: 1.047 s <<< FAILURE!
> java.lang.AssertionError
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertNotNull(Assert.java:712)
> at org.junit.Assert.assertNotNull(Assert.java:722)
> at
> org.apache.hadoop.yarn.server.timelineservice.security.TestTimelineAuthFilterForV2.verifyEntity(TestTimelineAuthFilterForV2.java:282)
> at
> org.apache.hadoop.yarn.server.timelineservice.security.TestTimelineAuthFilterForV2.testPutTimelineEntities(TestTimelineAuthFilterForV2.java:421)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runn
[jira] [Commented] (YARN-6929) yarn.nodemanager.remote-app-log-dir structure is not scalable
[
https://issues.apache.org/jira/browse/YARN-6929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16818738#comment-16818738
]
Prabhu Joseph commented on YARN-6929:
-
[~pbacsko] [~eyang] Have changed the app log dir structure to below and tested
on a test cluster. Can you review the latest patch when you get some time.
{code}
{aggregation_log_root}/{user}/bucket_{suffix}/{bucket1}/{appId}
where suffix is logs or logs-ifile
bucket1 is application#getId % 1
{code}
> yarn.nodemanager.remote-app-log-dir structure is not scalable
> -
>
> Key: YARN-6929
> URL: https://issues.apache.org/jira/browse/YARN-6929
> Project: Hadoop YARN
> Issue Type: Bug
> Components: log-aggregation
>Affects Versions: 2.7.3
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Major
> Attachments: YARN-6929-007.patch, YARN-6929-008.patch,
> YARN-6929-009.patch, YARN-6929-010.patch, YARN-6929.1.patch,
> YARN-6929.2.patch, YARN-6929.2.patch, YARN-6929.3.patch, YARN-6929.4.patch,
> YARN-6929.5.patch, YARN-6929.6.patch, YARN-6929.patch
>
>
> The current directory structure for yarn.nodemanager.remote-app-log-dir is
> not scalable. Maximum Subdirectory limit by default is 1048576 (HDFS-6102).
> With retention yarn.log-aggregation.retain-seconds of 7days, there are more
> chances LogAggregationService fails to create a new directory with
> FSLimitException$MaxDirectoryItemsExceededException.
> The current structure is
> //logs/. This can be
> improved with adding date as a subdirectory like
> //logs//
> {code}
> WARN
> org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService:
> Application failed to init aggregation
> org.apache.hadoop.yarn.exceptions.YarnRuntimeException:
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.protocol.FSLimitException$MaxDirectoryItemsExceededException):
> The directory item limit of /app-logs/yarn/logs is exceeded: limit=1048576
> items=1048576
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.verifyMaxDirItems(FSDirectory.java:2021)
>
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.addChild(FSDirectory.java:2072)
>
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.unprotectedMkdir(FSDirectory.java:1841)
>
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirsRecursively(FSNamesystem.java:4351)
>
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirsInternal(FSNamesystem.java:4262)
>
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirsInt(FSNamesystem.java:4221)
>
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirs(FSNamesystem.java:4194)
>
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.mkdirs(NameNodeRpcServer.java:813)
>
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.mkdirs(ClientNamenodeProtocolServerSideTranslatorPB.java:600)
>
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
>
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:619)
>
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:962)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2039)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2035)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
>
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2033)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService.createAppDir(LogAggregationService.java:308)
>
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService.initAppAggregator(LogAggregationService.java:366)
>
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService.initApp(LogAggregationService.java:320)
>
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService.handle(LogAggregationService.java:443)
>
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService.handle(LogAggregationService.java:67)
>
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:173)
>
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:106)
> at java.lang.Thread.run(Thread.java:745)
> Caused by:
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.protocol.FSLimitE
[jira] [Commented] (YARN-6695) Race condition in RM for publishing container events vs appFinished events causes NPE
[
https://issues.apache.org/jira/browse/YARN-6695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16818735#comment-16818735
]
Prabhu Joseph commented on YARN-6695:
-
[~eyang] Thanks for checking this. Have tried below few ways to change the
ordering which improves from consistent failure to intermittent but did not
find a better way to ensure collector removal only after all events handled as
they are handled asynchronously.
1. Send ATTEMPT_FINISHED to {{RMAppImpl}} after sending APP_ATTEMPT_REMOVED to
Scheduler in {{RMAppAttemptImpl}}
2. Stop Timeline Collector as part of doneApplication in Scheduler
Another approach is to remove the collectors after a configured collector
linger period similar to YARN-3995 done for NM Events. Can you check if this
approach is fine.
> Race condition in RM for publishing container events vs appFinished events
> causes NPE
> --
>
> Key: YARN-6695
> URL: https://issues.apache.org/jira/browse/YARN-6695
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Rohith Sharma K S
>Priority: Critical
> Attachments: YARN-6695.001.patch
>
>
> When RM publishes container events i.e by enabling
> *yarn.rm.system-metrics-publisher.emit-container-events*, there is race
> condition for processing events
> vs appFinished event that removes appId from collector list which cause NPE.
> Look at the below trace where appId is removed from collectors first and then
> corresponding events are processed.
> {noformat}
> 2017-06-06 19:28:48,896 INFO capacity.ParentQueue
> (ParentQueue.java:removeApplication(472)) - Application removed - appId:
> application_1496758895643_0005 user: root leaf-queue of parent: root
> #applications: 0
> 2017-06-06 19:28:48,921 INFO collector.TimelineCollectorManager
> (TimelineCollectorManager.java:remove(190)) - The collector service for
> application_1496758895643_0005 was removed
> 2017-06-06 19:28:48,922 ERROR metrics.TimelineServiceV2Publisher
> (TimelineServiceV2Publisher.java:putEntity(451)) - Error when publishing
> entity TimelineEntity[type='YARN_CONTAINER',
> id='container_e01_1496758895643_0005_01_02']
> java.lang.NullPointerException
> at
> org.apache.hadoop.yarn.server.resourcemanager.metrics.TimelineServiceV2Publisher.putEntity(TimelineServiceV2Publisher.java:448)
> at
> org.apache.hadoop.yarn.server.resourcemanager.metrics.TimelineServiceV2Publisher.access$100(TimelineServiceV2Publisher.java:72)
> at
> org.apache.hadoop.yarn.server.resourcemanager.metrics.TimelineServiceV2Publisher$TimelineV2EventHandler.handle(TimelineServiceV2Publisher.java:480)
> at
> org.apache.hadoop.yarn.server.resourcemanager.metrics.TimelineServiceV2Publisher$TimelineV2EventHandler.handle(TimelineServiceV2Publisher.java:469)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:201)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:127)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-6695) Race condition in RM for publishing container events vs appFinished events causes NPE
[
https://issues.apache.org/jira/browse/YARN-6695?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prabhu Joseph reassigned YARN-6695:
---
Assignee: Prabhu Joseph
> Race condition in RM for publishing container events vs appFinished events
> causes NPE
> --
>
> Key: YARN-6695
> URL: https://issues.apache.org/jira/browse/YARN-6695
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Rohith Sharma K S
>Assignee: Prabhu Joseph
>Priority: Critical
> Attachments: YARN-6695.001.patch
>
>
> When RM publishes container events i.e by enabling
> *yarn.rm.system-metrics-publisher.emit-container-events*, there is race
> condition for processing events
> vs appFinished event that removes appId from collector list which cause NPE.
> Look at the below trace where appId is removed from collectors first and then
> corresponding events are processed.
> {noformat}
> 2017-06-06 19:28:48,896 INFO capacity.ParentQueue
> (ParentQueue.java:removeApplication(472)) - Application removed - appId:
> application_1496758895643_0005 user: root leaf-queue of parent: root
> #applications: 0
> 2017-06-06 19:28:48,921 INFO collector.TimelineCollectorManager
> (TimelineCollectorManager.java:remove(190)) - The collector service for
> application_1496758895643_0005 was removed
> 2017-06-06 19:28:48,922 ERROR metrics.TimelineServiceV2Publisher
> (TimelineServiceV2Publisher.java:putEntity(451)) - Error when publishing
> entity TimelineEntity[type='YARN_CONTAINER',
> id='container_e01_1496758895643_0005_01_02']
> java.lang.NullPointerException
> at
> org.apache.hadoop.yarn.server.resourcemanager.metrics.TimelineServiceV2Publisher.putEntity(TimelineServiceV2Publisher.java:448)
> at
> org.apache.hadoop.yarn.server.resourcemanager.metrics.TimelineServiceV2Publisher.access$100(TimelineServiceV2Publisher.java:72)
> at
> org.apache.hadoop.yarn.server.resourcemanager.metrics.TimelineServiceV2Publisher$TimelineV2EventHandler.handle(TimelineServiceV2Publisher.java:480)
> at
> org.apache.hadoop.yarn.server.resourcemanager.metrics.TimelineServiceV2Publisher$TimelineV2EventHandler.handle(TimelineServiceV2Publisher.java:469)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:201)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:127)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
